cip-auditor: alerts are noisy

[spiffxp]

Tracking issue for the fact that CIP auditor alerts are noisy.

The alerting was manually created by thockin a while ago (could not find an issue to link at a glance, maybe someone else knows). Then it was manually disabled because the alerts were perceived as noisy.

The alerting is managed via click-ops, ideally it could be done with gcloud: #1624

IAM policies to allow viewing of incidents and alerts without granting admin access to k8s-artifacts prod would be helpful. What's the appropriate group to grant this access to?

/cc @listx
FYI @tylerferrara
Making this tracking issue since I'm about to make a manual IAM change and I want to document it somewhere

[spiffxp]

/priority important-longterm
/wg k8s-infra
/sig release
/area artifacts
/area release-eng
/area infra/monitoring
/milestone v1.22

[spiffxp]

FWIW here is the policy generating the alert in question

# $ gcloud alpha monitoring policies list --project=kubernetes-public
---
combiner: OR
conditions:
- conditionThreshold:
    aggregations:
    - alignmentPeriod: 60s
      crossSeriesReducer: REDUCE_MAX
      perSeriesAligner: ALIGN_RATE
    comparison: COMPARISON_GT
    duration: 0s
    filter: metric.type="logging.googleapis.com/user/cip-auditor-alert"
    trigger:
      count: 1
  displayName: logging/user/cip-auditor-alert
  name: projects/kubernetes-public/alertPolicies/12089518252001301876/conditions/12089518252001300531
creationRecord:
  mutateTime: '2020-03-31T00:27:12.391234443Z'
  mutatedBy: thockin@REDACTED
displayName: Image promoter alert
documentation:
  content: The image promoter has logged something that we consider an alert.
  mimeType: text/markdown
enabled: true
mutationRecord:
  mutateTime: '2020-03-31T01:45:19.410085124Z'
  mutatedBy: thockin@REDACTED
name: projects/kubernetes-public/alertPolicies/12089518252001301876
notificationChannels:
- projects/kubernetes-public/notificationChannels/7630148271419930225
- projects/kubernetes-public/notificationChannels/17367851054639804370
- projects/kubernetes-public/notificationChannels/2533614711603005061
- projects/kubernetes-public/notificationChannels/7846745591716920888

[tylerferrara]

/assign

[spiffxp]

members of k8s-infra-gcp-auditors should be able to do what I just did above:

k8s.io/infra/gcp/roles/audit.viewer.yaml

Lines 882 to 883 in 15f7d7c

	- monitoring.alertPolicies.get
	- monitoring.alertPolicies.list

[spiffxp]

This is what IAM is for the project in question right now: https://github.com/kubernetes/k8s.io/blob/main/audit/projects/kubernetes-public/iam.json

I could be convinced that k8s-infra-cluster-admins should get something like roles/monitoring.editor or higher on this project, as the nearest conveniently existing group that has editor/admin type roles on this project

I could also be convinced there should be a group dedicated to CIP related infrastructure that gets more granular IAM permissions

[spiffxp]

@tylerferrara is unable to view incidents because of lack of roles/monitoring.viewer, ref: https://cloud.google.com/monitoring/alerts/troubleshooting-alerts#no-permission

The only existing group that has this is gke-security-groups:

k8s.io/audit/projects/kubernetes-public/iam.json

Lines 117 to 122 in 15f7d7c

	"members": [
	"group:gke-security-groups@kubernetes.io",
	"serviceAccount:gke-nodes-aaa@kubernetes-public.iam.gserviceaccount.com",
	"serviceAccount:k8s-infra-monitoring-viewer@kubernetes-public.iam.gserviceaccount.com"
	],
	"role": "roles/monitoring.viewer"

Which is populated entirely by app-specific rbac groups:

k8s.io/groups/groups.yaml

Lines 172 to 194 in 15f7d7c

	# Every RBAC group should be added here.
	- email-id: gke-security-groups@kubernetes.io
	name: gke-security-groups
	description: |-
	Security Groups for GKE clusters
	settings:
	ReconcileMembers: "true"
	WhoCanViewMembership: "ALL_MEMBERS_CAN_VIEW" # needed for RBAC
	members:
	- k8s-infra-rbac-cert-manager@kubernetes.io
	- k8s-infra-rbac-gcsweb@kubernetes.io
	- k8s-infra-rbac-kettle@kubernetes.io
	- k8s-infra-rbac-k8s-io-canary@kubernetes.io
	- k8s-infra-rbac-k8s-io-prod@kubernetes.io
	- k8s-infra-rbac-kubernetes-external-secrets@kubernetes.io
	- k8s-infra-rbac-perfdash@kubernetes.io
	- k8s-infra-rbac-prow@kubernetes.io
	- k8s-infra-rbac-publishing-bot@kubernetes.io
	- k8s-infra-rbac-triageparty-release@kubernetes.io
	- k8s-infra-rbac-triageparty-scalability@kubernetes.io
	- k8s-infra-rbac-sippy@kubernetes.io
	- k8s-infra-rbac-slack-infra@kubernetes.io
	- k8s-infra-rbac-node-perf-dash@kubernetes.io

Rather than untangle the right place to put them, since I'm about to go AFK, I'm going to add @tylerferrara manually

$ gcloud projects add-iam-policy-binding kubernetes-public --role=roles/monitoring.viewer --member=user:tjferrara@google.com
Updated IAM policy for project [kubernetes-public].