-
Notifications
You must be signed in to change notification settings - Fork 1.2k
feature: saving files as blob in database #1326
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
some clean up tasks etc are needed. just wanted to see if there is interest in a feature like this. |
Although it is possible to save files in the database, we do not recommend it. Since the number of files can be large, saving them in the database not only causes performance problems with queries, it can also be very troublesome when it comes time to back up database data. You should be able to find some bad things about storing files in databases in the web. All in all we still don't recommend users to store files in the database. Storing them in the file system or a third-party cloud provider using a plugin is good. |
+1 for this feature. It is quite troublesome to keep backups and restores in sync if you have multiple infrastructure components. It's also a cost aspect for us. |
+1 for this PR I would also like to have that ability as we are using Apache Answers also. At the moment we have to align the backups of the filesystem AND the database backup as well. If we use the ability to store the files as blobs within the database, we would only have to care about a single backup component. We could use the toggle that @DanielAuerX already provided to explicitly enable the blob feature, by default we simply set FILE_STORAGE_MODE to 'fs'. What do you say @LinkinStars ? :) Cheers, |
Thanks for the feedback, it's important for us to hear more varied suggestions. Of course, there are advantages and disadvantages to putting files in the database. I understand that this can be turned off by default by setting ENV to keep the existing feature unaffected. Other than that, we'll mainly be looking at maintenance costs. Let's hear more ideas from others before we decide on this feature. @sy-records @kumfo |
+ files (e.g. branding pictues, attachments) can be saved as a blob in a database. The feature can be turned on and off (feature toggle) with the env var FILE_STORAGE_MODE + a new controller with the endpoints /file/branding etc. had to be created since the files are statically linked when fs is used. + the files plus metadata are stored in the table 'file'
When I was looking into how files are deleted, I noticed some issues with the file_record table. This is actually related to the problem of keeping the file system and the db in sync.
I might missed or misunderstood some things, but I am quite sure there is something wrong here |
@DanielAuerX Thanks for the feedback.
|
Yes sure, I will prepare a PR |
Thank you for your contribution. I also think it would add extra maintenance costs, after all, most use local storage or 3rd party cloud storage. If this can exist as a plugin, that's good. |
- [x] create file_record table - [x] avatar and branding files are added to file_record - [x] branding files are being deleted - [x] avatar files are being deleted - [x] reload latest avatar (frontend) after backend state is being updated problems addressed in the pr: - clean up job fails, because it cannot access file_record table - avatar and branding files are not added to the file_record table - avatar and branding files are never deleted - after an avatar is being updated/deleted, the old file is still being requested due to browser caching. This is causing error logs ("no such file or directory") in the backend. cf. conversation in [pr 1326](#1326) --------- Co-authored-by: broccoli <[email protected]> Co-authored-by: LinkinStars <[email protected]>
For me it's not about the extra maintenance costs - but the extra infrastructure costs that we would have to pay when we have to use local storage (persistent volumes in k8s) and a postgres database. It's a lot easier to maintain and reduces production costs if we only have to use the (already existing) postgres database. Adding S3 or even local storage is not that easy in cluster environments. |
files (e.g. branding pictues, attachments) can be saved as a blob in a database. The feature can be turned on and off (feature toggle) with the env var FILE_STORAGE_MODE
a new controller with the endpoints /file/branding etc. had to be created since the files are statically linked when fs is used.
the files plus metadata are stored in the table 'file'