[BUG] - 409 error for keycloak on deployment (we are managing keycloak outside of tf?)

[asmacdo]

Describe the bug

I think this is caused by importing from backup/syncing users with keycloak directly. IMO it is better to do that directly with keycloak, but I'd like to not have to delete those groups, then restore keycloak from backup each deploy

output

[tofu]: data.keycloak_role.view-users: Reading...
[tofu]: data.keycloak_role.query-users: Reading...
[tofu]: data.keycloak_role.manage-users: Reading...
[tofu]: data.keycloak_role.query-groups: Reading...
[tofu]: data.keycloak_role.realm-admin: Reading...
[tofu]: data.keycloak_role.query-users: Read complete after 0s [id=24de488a-8128-4fd0-965d-39318a2863c0]
[tofu]: data.keycloak_role.view-users: Read complete after 0s [id=92ca1e83-c7a5-4684-b94d-7772eae16903]
[tofu]: keycloak_user_roles.user_roles: Refreshing state... [id=master/0b1d3e95-425e-4a76-b7d4-d0f71e6f27dc]
[tofu]: data.keycloak_role.manage-users: Read complete after 0s [id=8b226316-921e-4e9c-83e1-d95341c8de90]
[tofu]: data.keycloak_role.realm-admin: Read complete after 0s [id=67e31a4f-64df-48df-ba27-e244772bc666]
[tofu]: keycloak_group_roles.superadmin_roles: Refreshing state... [id=nebari/b1e7907e-b16f-46ff-b85e-fc7a7d2905e5]
[tofu]: data.keycloak_role.query-groups: Read complete after 0s [id=d1917caa-cfbd-448f-8dd3-0aa75b279628]
[tofu]: keycloak_group_roles.admin_roles: Refreshing state... [id=nebari/1cd3d1b8-f248-4bf0-a35a-c5727fb70183]
[tofu]: 
[tofu]: Note: Objects have changed outside of OpenTofu
[tofu]: 
[tofu]: OpenTofu detected the following changes made outside of OpenTofu since the
[tofu]: last "tofu apply" which may have affected this plan:
[tofu]: 
[tofu]:   # keycloak_group.groups["superadmin"] has been deleted
[tofu]:   - resource "keycloak_group" "groups" {
[tofu]:       - id       = "b1e7907e-b16f-46ff-b85e-fc7a7d2905e5" -> null
[tofu]:         name     = "superadmin"
[tofu]:         # (2 unchanged attributes hidden)
[tofu]:     }
[tofu]: 
[tofu]: 
[tofu]: Unless you have made equivalent changes to your configuration, or ignored the
[tofu]: relevant attributes using ignore_changes, the following plan may include
[tofu]: actions to undo or respond to these changes.
[tofu]: 
[tofu]: ─────────────────────────────────────────────────────────────────────────────
[tofu]: 
[tofu]: OpenTofu used the selected providers to generate the following execution
[tofu]: plan. Resource actions are indicated with the following symbols:
[tofu]:   + create
[tofu]:   ~ update in-place
[tofu]: 
[tofu]: OpenTofu will perform the following actions:
[tofu]: 
[tofu]:   # keycloak_default_groups.default will be updated in-place
[tofu]:   ~ resource "keycloak_default_groups" "default" {
[tofu]:       ~ group_ids = [
[tofu]:           - "213bde14-004d-4778-9b5b-0a1e315cad2f",
[tofu]:         ] -> (known after apply)
[tofu]:         id        = "nebari/default-groups"
[tofu]:         # (1 unchanged attribute hidden)
[tofu]:     }
[tofu]: 
[tofu]:   # keycloak_group.groups["analyst"] will be created
[tofu]:   + resource "keycloak_group" "groups" {
[tofu]:       + id       = (known after apply)
[tofu]:       + name     = "analyst"
[tofu]:       + path     = (known after apply)
[tofu]:       + realm_id = "nebari"
[tofu]:     }
[tofu]: 
[tofu]:   # keycloak_group.groups["developer"] will be created
[tofu]:   + resource "keycloak_group" "groups" {
[tofu]:       + id       = (known after apply)
[tofu]:       + name     = "developer"
[tofu]:       + path     = (known after apply)
[tofu]:       + realm_id = "nebari"
[tofu]:     }
[tofu]: 
[tofu]:   # keycloak_group.groups["superadmin"] will be created
[tofu]:   + resource "keycloak_group" "groups" {
[tofu]:       + id       = (known after apply)
[tofu]:       + name     = "superadmin"
[tofu]:       + path     = (known after apply)
[tofu]:       + realm_id = "nebari"
[tofu]:     }
[tofu]: 
[tofu]:   # keycloak_group_roles.superadmin_roles will be created
[tofu]:   + resource "keycloak_group_roles" "superadmin_roles" {
[tofu]:       + exhaustive = false
[tofu]:       + group_id   = (known after apply)
[tofu]:       + id         = (known after apply)
[tofu]:       + realm_id   = "nebari"
[tofu]:       + role_ids   = [
[tofu]:           + "67e31a4f-64df-48df-ba27-e244772bc666",
[tofu]:         ]
[tofu]:     }
[tofu]: 
[tofu]: Plan: 4 to add, 1 to change, 0 to destroy.
[tofu]: keycloak_group.groups["developer"]: Creating...
[tofu]: keycloak_group.groups["superadmin"]: Creating...
[tofu]: keycloak_group.groups["analyst"]: Creating...
[tofu]: ╷
[tofu]: │ Error: error sending POST request to /auth/admin/realms/nebari/groups: 409 Conflict. Response body: {"errorMessage":"Top level group named 'analyst' already exists."}
[tofu]: │ 
[tofu]: │   with keycloak_group.groups["analyst"],
[tofu]: │   on main.tf line 71, in resource "keycloak_group" "groups":
[tofu]: │   71: resource "keycloak_group" "groups" {
[tofu]: │ 
[tofu]: ╵
[tofu]: ╷
[tofu]: │ Error: error sending POST request to /auth/admin/realms/nebari/groups: 409 Conflict. Response body: {"errorMessage":"Top level group named 'developer' already exists."}
[tofu]: │ 
[tofu]: │   with keycloak_group.groups["developer"],
[tofu]: │   on main.tf line 71, in resource "keycloak_group" "groups":
[tofu]: │   71: resource "keycloak_group" "groups" {
[tofu]: │ 
[tofu]: ╵
[tofu]: ╷
[tofu]: │ Error: error sending POST request to /auth/admin/realms/nebari/groups: 409 Conflict. Response body: {"errorMessage":"Top level group named 'superadmin' already exists."}
[tofu]: │ 
[tofu]: │   with keycloak_group.groups["superadmin"],
[tofu]: │   on main.tf line 71, in resource "keycloak_group" "groups":
[tofu]: │   71: resource "keycloak_group" "groups" {
[tofu]: │ 
[tofu]: ╵

Expected behavior

should be ok if group already exists

OS and architecture in which you are running Nebari

fedora locally, aws for cloud

How to Reproduce the problem?

Not entirely sure, seems to be restore keycloak from backup, then attempt to redeploy.

Command output

Versions and dependencies used.

No response

Compute environment

None

Integrations

No response

Anything else?

No response

[viniciusdc]

This is an issue with a mismatch between the keycloak DB data/resources vs what the terraform expects in its state. To better address this now, you would need to import the resources into the tf state using https://registry.terraform.io/providers/mrparkers/keycloak/latest/docs/resources/role#import.

You would need to run the command for each component that's missing in the state, there is also a list command to show what's available already. To see the uids for the resources you might need to do some digging inside the docs for the provider above just to make sure what's the values you need to pass as UIDs etc.

This is a good finding, I will will try to document this in our docs as part of the backup & restore process

[viniciusdc]

Another thing you can do, is delete the groups resources tf is complaining about, and use the keycloak API with PUT methods to override the resources with the data from your backup

[asmacdo]

I hit this again yesterday, had 409s for analyst, developer, and superadmin on nebari deploy. I manually deleted these groups, and let nebari recreate them. The difference is that now, rather than restoring user/group mapping from backup, I have a script that uses the keycloak python API to add/modify them. I'm hoping that this route will avoid the issue because we are no longer restoring from a backup, hopefully there will not be that terraform drift in the uuids as mentioned by @viniciusdc.

However, I'd still consider this a workaround, IIUC the actual problem is that if we use the backup/restore workflow with keycloak, we can no longer run nebari deploy without manual intervention.

[viniciusdc]

@asmacdo, this is the backup workflow you are following for keycloak https://nebari.dev/docs/how-tos/manual-backup#keycloak-usergroup-database? I think that might need some updates. I will try to run this tomorrow and push some fixes.

Overall, I think I understood the problem you were facing, but I don't know why keycloak is trying to re-create the groups every time. Are you making any changes to the state before the re-deploy or deleting the groups/postgresDB? Or are you destroying the cluster every time?

If you could describe the process in a list, that would help me replicate it as well. Many thanks 🚀