Add possibility to configure wildcard namespace value in Alert eventSources

[anders-olofsson]

It would be nice to pick up all error events within a cluster using a wildcard namespace value in the eventSources section of an Alert - similar to how it works for name:

notification-controller/internal/server/event_handlers.go

Line 77 in fbf1ea0

if (source.Name == "*" || event.InvolvedObject.Name == source.Name) &&

Currently you'll either need to duplicate Provider (including any secrets) and Alert resources to all relevant namespaces or create a "global" Alert which references other namespaces. It would be simpler and less error prone to allow namespace: "*".

[phillebaba]

I dont see any issue implementing this, as it would just be a catch all for all events in all namespaces. Or events from a specific resource kind in all namespaces. As we already allow event sources from multiple namespaces it would just act as a helper to avoid having to type out each namespace. One thing to keep in mind is that there would probably be a lot of events generated from doing this.

@stefanprodan do you have any opinion about this?

[stefanprodan]

This would break multi-tenancy, imagine a tenant will create a "global" alert in their namespace and it will receive events with sensitive information about all the other tenants. I find this unacceptable, imagine if AWS would allow anyone to route all events from Cloudwatch no matter the account.

[phillebaba]

I have also been thinking about that. In theory it would be possible if we were able to limit the event sources to the permissions the service account that is assumed has. We can obviously not allow anyone to export all events in the cluster.

[anders-olofsson]

Not sure if it helps, but it'd be fine if it was only allowed for alerts defined in the flux-system namespace.

[Niksko]

Not sure if it helps, but it'd be fine if it was only allowed for alerts defined in the flux-system namespace.

This would work for us.

That said, I think the issue around multi-tenancy is a cluster policy issue rather than something that needs to be solved by this project. It's up to the cluster administrator to set policy on who can create Alert resources, and whether they can create them with a namespace of *.

Perhaps as a stopgap, the ability to have * for the namespace could be behind a flag, so that the cluster administrators have a simple way to allow or disallow this functionality?

[nvanheuverzwijn]

How about a new kind, ClusterAlert, in order to be alerted on all object clusterwide ? Like ClusterRole and Role.

[arash-bizcover]

This would break multi-tenancy, imagine a tenant will create a "global" alert in their namespace and it will receive events with sensitive information about all the other tenants. I find this unacceptable, imagine if AWS would allow anyone to route all events from Cloudwatch no matter the account.

This is no way equivilant comparison. We allow flux in a single namespace(flux-system) to manage resources in all other namespaces without explicitly opening up each namespace for flux controllers. In the same approach, we should be able to set notifications+alert for those resources.

[itspngu]

I, as many others, would have a use case for the proposed behaviour, and would be happy to put some work towards it. I've used Flux extensively for about a year now but never really dug into the codebase.

For what it's worth I feel like @nvanheuverzwijn's suggestion of having a non-namespaced ClusterAlert resource is the cleaner approach, because it doesn't break any existing functionality, accurately represents what namespace: '*' would try to achieve on a more abstract level, and would allow using RBAC and other familiar mechanisms for multi-tenancy scenarios.

Could this issue be put onto the agenda for the next meeting or something? It seems that there's some disagreement about if and how this should be implemented at all, and I'd like to have a clear goal for what a PR solving this issue should entail.

[stefanprodan]

A ClusterAlert doesn't solve much because it would refer to a ClusterProvider that would refer to a Kubernetes secret, and Kubernetes team rejected the proposal of having ClusterSecrets. I'm for revisiting the namespace wildcard option after RFC-0003 gets approved.