[NGF][2.1.0] Mounting cert as secret has odd behavior when applying nginx config

[michasHL]

Hello NGF team,

we got another odd case where NGF is having problems applying the NGF config.
We're mounting a root certificate into the data plane deployment by using the following values in the helm chart:

nginx:
  pod:
    volumes:
    - name: root-ca
        secret:
          secretName: secret
  container:
    volumeMounts:
    - name: root-ca
        mountPath: /etc/nginx/certs/

The SnippetsFilter looks like this:

apiVersion: gateway.nginx.org/v1alpha1
kind: SnippetsFilter
metadata:
  name: test
spec:
  snippets:
  - context: http.server
    value: |
      client_header_buffer_size 2k;
      ssl_client_certificate /etc/nginx/certs/root-ca.crt;
      ssl_verify_client optional;
      ssl_verify_depth 2;

      proxy_connect_timeout 60s;
      proxy_read_timeout 60s;
      proxy_send_timeout 60s;
  - context: http.server.location
    value: "proxy_set_header x-ssl-client-cert $ssl_client_escaped_cert;\n\nlocation
      ~ ^(?!(/status/live|/canary/status/live|/instance/.+/status/live|/status/ready|/canary/status/ready|/instance/.+/status/ready))
      { \n  if ($ssl_client_verify != SUCCESS) {\n    return 403;\n  }\n}\n\nset $smp
      \"${request_method}${uri}\";\nif ( $smp ~* ^(?!(GET/status/live|GET/canary/status/live|GET/instance/.+/status/live|GET/status/ready|GET/canary/status/ready|GET/instance/.+/status/ready|))
      ) {\n  return 403;\n}\n"

After rolling the control plane, the gateway goes into programmed state for a while. It seems upon the first change we're seeing this in the Gateway resource:
Message: The Gateway is not programmed due to a failure to reload nginx with the configuration: msg: Config apply failed, rolling back config; error: error deleting file: /etc/nginx/certs/root-ca.crt error: remove /etc/nginx/certs/root-ca.crt: read-only file system

Now of course, the simple answer might be to just move the cert to a different folder, because maybe it doesn't like that the cert file is mounted to /etc/nginx/. Otherwise, I'm a little baffled because secret volumes always seem to be added as read-only: https://kubernetes.io/docs/concepts/storage/volumes/#secret.

[sjberman]

I think the reason that the NGINX Agent is trying to remove the file is because it's not being provided directly by our control plane (since you're mounting it), and any file not provided by our control plane will be cleaned up unless we explicitly tell agent to ignore it. We'd need to implement a way to add any mounted files into that ignore list.

[sjberman]

I'll create an issue for this to see how we could accomplish it. I think it's doable.

[sjberman]

#3776

[michasHL]

Thank you. Very much appreciated.