Support for writing Grafana Service Account token Secrets into a custom namespace

[belmin-devops-eng]

Summary

Currently, when the Grafana Operator creates a Grafana Service Account, the associated token Secret is created in the same namespace as the Grafana instance. This works fine when other components (e.g., Kiali) that need to integrate with Grafana are deployed in the same namespace.

However, if Grafana is deployed in a different namespace than Kiali, the integration fails because Kiali cannot access the token Secret from another namespace.

This limitation makes cross-namespace integrations difficult and forces users to co-locate components in the same namespace, which is not always desirable for organizational or security reasons.

Use Case

• Grafana is deployed in namespace grafana-ns.
• Kiali is deployed in namespace observability.
• Kiali needs to access Grafana using the Grafana Service Account token.
• Currently, the token Secret is created in grafana-ns, so Kiali cannot access it.

If the operator supported writing the token Secret into a custom namespace (e.g., observability), Kiali could access it without requiring both components to share the same namespace.

Proposed Solution

Add a configurable target namespace for generated token Secrets on the GrafanaServiceAccount CRD.
This should be per-token for flexibility.

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaServiceAccount
metadata:
  name: my-grafana-sa
  namespace: grafana-ns
spec:
  name: grafana-kiali-integration
  role: Viewer
  instanceName: grafana-cr
  isDisabled: false
  tokens:
    - name: grafana-kiali-serviceaccount-token
      secretName: grafana-kiali-serviceaccount-token-secret
      secretNamespace: observability    # NEW: write Secret into this namespace

Notes

• Operator will need RBAC permissions to create/update/delete Secrets in the target namespace.
• If secretNamespace is omitted, behavior remains unchanged (Secret stays in the GSA namespace).
• Clear error/status should be surfaced if the operator lacks permission to write in the target namespace.

[theSuess]

Thanks for the report. We'll look into implementing this.

One complication that came to mind when discussing the implementation was the absence of cross-namespace owner references for garbage collection. Nothing that can't be solved but we'll need to keep it in mind when building.

If you're open to contribute this functionality, we're happy to review any PRs for this!

[hkmdxlftjf]

@theSuess hi. I can work on this

[hkmdxlftjf]

I feel that using this approach is better. The GrafanaServiceAccount is created in another namespace, and it can bind to a Grafana CR located in a different namespace.

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaServiceAccount
metadata:
  name: my-grafana-sa
  namespace: observability    # write Secret into this namespace
spec:
  name: grafana-kiali-integration
  role: Viewer
  instanceName: grafana-cr
  instanceNamespace: grafana-cr-namespace  # NEW: write Secret into this namespace
  isDisabled: false
  tokens:
    - name: grafana-kiali-serviceaccount-token
      secretName: grafana-kiali-serviceaccount-token-secret

[Baarsgaard]

Limiting the GrafanaServiceAccount to the same namespace as the instance is a deliberate design choice 🙂
We discussed this a lot and you can read about it in the design proposal document

Workarounds exist, like having External Grafana CRs in multiple namespaces.
But we intend to keep the .spec.instanceName limitation and only allow token secrets to cross namespace boundaries.

And even so, we need to make sure we do not overwrite existing secrets.

[belmin-devops-eng]

Big thanks for the support, guys! Closing the issue now.

[theSuess]

Keeping this open since it's something we want to implement eventually