Safe proxy buffer configuration

[danielnginx]

I want NIC to provide safety around the relationship of proxy_buffers and proxy_buffer_size
Since modifying one of these settings impacts the other, the system should automatically adjust the 'other' setting to ensure that any new pods can start.

Explore how ingress-nginx solved this problem, the solution does not need to be more complicated nor precise.
This is about ensuring out of the box success and not expecting deep knowledge with NGINX.

Things to do:

• Setting nginx 'proxy-buffer-size' to 16k causes nginx config error kubernetes/ingress-nginx#363 needs to be implemented in NIC
• a new annotation, VS and configmap option needs to be exposed for proxy_busy_buffers_size, see nginx validation here
• "there must be at least 2 "proxy_buffers""); setting 1 proxy buffer results in nginx reload error: verify if this is the case and if yes put validation for corresponding annotation, vs and cm option
• Upgrade tests
• Docs

nginx.org/proxy-buffer-size: "5m"
nginx.org/proxy-buffers: "8 1m"

[vepatel]

blocked as no annotation exists for directive proxy_busy_buffers_size in NIC

[brianehlert]

first outcome: automatically tune proxy_busy_buffers_size in relation to the related directives being changed to ensure the settings are always compatible.
This is to ensure that proxy_busy_buffers_size or proxy_buffers or proxy_buffers_size set incorrectly in relation to each other doesn't block a NIC pod from starting.
Since at startup, the configuration defined in the K8s API is required to be valid, or NIC pods will never become ready because NGINX determines the configuration to be invalid.

I am not opinionated about exposing proxy_busy_buffers_size directly. However, I do not see an apparent advantage.
I fear that exposing it as its own knob adds additional complexity for the customer and system.

[AlexFenlon]

Ref:

• Setting nginx 'proxy-buffer-size' to 16k causes nginx config error kubernetes/ingress-nginx#363
• Missing annotation for proxy_busy_buffers_size kubernetes/ingress-nginx#9356
• https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_busy_buffers_size