I'm running into a super strange issue with /run/secrets not working

[dkowis]

I suspect it may have something to do with recent changes that mention moving this directory around:

• Bug: activation messages get confused about rendered templates #652

My problem is that when the system reboots, I don't get a proper /run/secrets directory that is a symlink to the right location.
I have a directory /run/secrets/rendered instead of it being a symlink to /run/secrets.d/1 with a rendered directory in it?

This host has an early secrets-for-users in a template, that probably doesn't need to be in a template, but I think that's what's happening.

I get this error on activation:

[root@sarafan:/etc/nixos]# nixos-rebuild switch
building the system configuration...
activating the configuration...
sops-install-secrets: Imported /etc/ssh/ssh_host_rsa_key as GPG key with fingerprint 8205c05ea62380d8950426c019a42470335483ff
sops-install-secrets: Imported /etc/ssh/ssh_host_ed25519_key as age key with fingerprint age18g8r7drww22ulsw3728n4nz9g5kcn0tvvvsw0dql89kkufttxe3s72j6fj
setting up /etc...
sops-install-secrets: Imported /etc/ssh/ssh_host_rsa_key as GPG key with fingerprint 8205c05ea62380d8950426c019a42470335483ff
sops-install-secrets: Imported /etc/ssh/ssh_host_ed25519_key as age key with fingerprint age18g8r7drww22ulsw3728n4nz9g5kcn0tvvvsw0dql89kkufttxe3s72j6fj
/nix/store/aax7zp4j2pi18mnig3xmzb64j9axvmww-sops-install-secrets-0.0.1/bin/sops-install-secrets: failed to prepare new secrets directory: cannot access /run/secrets: readlink /run/secrets: invalid argument
Activation script snippet 'setupSecrets' failed (1)
reloading user units for nixboi...
restarting sysinit-reactivation.target
warning: the following units failed: mysql.service, postgresql.service

And since my ssl key is sops'd it doesn't show up. So far, the only solution I've had is to go back in time generation wise, built on 9 Nov 2024.

[dkowis]

Yep, confirmed. If I change that template to not be neededForUsers, everything shows up fine.

I think it's an order of operations with the changes for the template directories rendering.

Two things come to mind for me:
A: I shouldn't have been doing neededForUsers on that.
B: It probably shouldn't footgun so hard.

Thanks!

[DrymarchonShaun]

I'm also seeing the same error, and I don't have any templates using neededForUsers. It seems #655 was when it started.

EDIT - Actually, I don't think you even can specify whether a template is available for users, I think they always are? (or at least they always show up in /run/secrets-for-users?)

EDIT 2 - To clarify what I'm actually seeing, this is the difference:

Before updating

$ ls -l /run/
lrwxrwxrwx     - root    10 Nov 15:26 secrets -> /run/secrets.d/1
lrwxrwxrwx     - root    10 Nov 15:26 secrets-for-users -> /run/secrets-for-users.d/2
drwxr-x--x     - root    10 Nov 15:26 secrets-for-users.d
drwxr-x--x     - root    10 Nov 15:26 secrets.d

$sudo ls -l /run/secrets/ 
total 4
-r--r--r-- 1 root root 93 Nov 10 15:37 github-token
drwxr-x--x 2 root keys  0 Nov 10 15:37 networks
drwxr-x--x 2 root keys  0 Nov 10 15:37 rendered
drwxr-x--x 2 root keys  0 Nov 10 15:37 ssh_keys
drwxr-x--x 2 root keys  0 Nov 10 15:37 user_age_keys

After updating and rebooting

$ ls -l /run/
drwxr-xr-x     - root    10 Nov 15:27 secrets
lrwxrwxrwx     - root    10 Nov 15:27 secrets-for-users -> /run/secrets-for-users.d/1
drwxr-x--x     - root    10 Nov 15:27 secrets-for-users.d
drwxr-x--x     - root    10 Nov 15:27 secrets.d

$ ls -l /run/secrets
drwxr-xr-x - root 10 Nov 15:27 rendered

[jfly]

Actually, I don't think you even can specify whether a template is available for users, I think they always are? (or at least they always show up in /run/secrets-for-users?)

I don't think sops-nix has support for templates for users. At least, I don't see an option here: https://github.com/Mic92/sops-nix/blob/f1675e3b0e1e663a4af49be67ecbc9e749f85eb7/modules/sops/templates/default.nix.

/nix/store/aax7zp4j2pi18mnig3xmzb64j9axvmww-sops-install-secrets-0.0.1/bin/sops-install-secrets: failed to prepare new secrets directory: cannot access /run/secrets: readlink /run/secrets: invalid argument

This is failing here:

sops-nix/pkgs/sops-install-secrets/main.go

Line 372 in f1675e3

linkTarget, err := os.Readlink(linkName)

, where we assume that /run/secrets (and /run/secrets-for-users) are symlinks. I can see in @DrymarchonShaun's output that /run/secrets is a folder.

So, the real question is "why is /run/secrets a folder"?

Since @DrymarchonShaun said this started happening after #655, I'm suspicious of this call to os.MkdirAll. The default path for templates is something like /run/secrets/rendered/foo, which probably never matches the if targetFile == template.Path (aka if "/run/secrets.d/1/rendered/foo" == "/run/secrets/rendered/foo") that's supposed to prevent us from calling os.MkdirAll("/run/secrets/rendered").

• I'm not sure why the existing tests aren't catching this. I'll work on reproducing it.
• I'm not sure what the right fix is yet. Reverting fix: create template.path symlink #655 isn't an option, because it's fixing a regression introduced by Templates refactoring #649.
• I don't know yet if we have a similar bug with the analogous if statement for secrets.

Edit: Nevermind, that theory was wrong. The if targetFile == template.Path does behave as it's supposed to.

[jfly]

I poked at this a bit, and I was able to reproduce a similar error if I set the path for one of my templates to be inside of /run/secrets. See #660.

@DrymarchonShaun @dkowis, is it possible that you have secrets or templates where you've configured their paths to be somewhere other than the default location in /run/secrets?

[DrymarchonShaun]

is it possible that you have secrets or templates where you've configured their paths to be somewhere other than the default location in /run/secrets?

No, I'm not setting the path at all. I take that back, I missed that you said secrets or templates. I do set the path of a secret just above where I setup the templates.

If you want to look at what I'm doing, I configure them templates here, and then I'm referencing them by their path like this.

[Mic92]

Does someone has a minimal configuration that can be reproduced in a nixos test or
https://github.com/Mic92/nixos-shell ? Having that should allow us to fix this quickly.

[dkowis]

I poked at this a bit, and I was able to reproduce a similar error if I set the path for one of my templates to be inside of /run/secrets. See #660.

@DrymarchonShaun @dkowis, is it possible that you have secrets or templates where you've configured their paths to be somewhere other than the default location in /run/secrets?

Not on the host that was having the troubles. I had changed this file to not use secretsForUsers. And then my directory was a symlink again.

the host I saw this problem on, was this one Other hosts, like keycloak were unaffected, and had proper symlinks, and the only difference I could find was the backup.nix file, which was using secretsForUsers unnecessarily.

A quick rg for template on my repo shows this:

~/g/nixos-configuration on  main via ❄️  impure (nix-shell-env)
❯ rg template *
hosts/traefik/default.nix
89:  sops.templates."traefik_environment.env" = {
134:            config.sops.templates."traefik_environment.env".path

hosts/registry/default.nix
51:  sops.templates."zot-config.json" = let
166:    source = config.sops.templates."zot-config.json".path;

hosts/sarafan/backup_scripts/postgresql_backup.sh
26:export FULL_BACKUP_QUERY="select datname from pg_database where not datistemplate and datallowconn order by datname;"

hosts/sarafan/default.nix
83:  # have to create the config file for the mysql exporter using templates and credentials.
84:  sops.templates."mysql-monitor.cfg" = {
96:  sops.templates."prometheus_exporter.yml" = {
107:      configFile = config.sops.templates."mysql-monitor.cfg".path; #This is the autogenerated path from the mysql package...

hosts/photos/default.nix
28:  sops.templates."immich.env" = {
48:  sops.templates."postgres.env" = {
94:            config.sops.templates."immich.env".path
126:            config.sops.templates."immich.env".path
144:            config.sops.templates."immich.env".path
170:            config.sops.templates."postgres.env".path

I looked into those finds, and none of them are setting the actual template path.

I'll try to find time to make a minimal reproduction.

[neumantm]

I have the same problem.
I quickly built a small example here: https://github.com/neumantm/minimal_example_for_sops_nix_659
Excuse my mediocre proficiency in nix and the resulting incomplete simplification.

I have noticed, that setting neededForUsers on some secret is actually required to reproduce the issue (or at least I failed to do that otherwise), hence the users.nix.

[bbigras]

I had the same problem. I think I fixed it by disabling all the services using the secrets and rebooting. Maybe only rebooting was required.

[jfly]

@neumantm, thanks for putting together a repro! Unfortunately, I'm not actually seeing anything wrong when I boot up your demo vm:

$ nixos-shell --flake github:neumantm/minimal_example_for_sops_nix_659
...

[root@nixos:~/src/github.com/neumantm/minimal_example_for_sops_nix_659]# ls -alh /run
...
lrwxrwxrwx  1 root root   16 Nov 11 19:42 secrets -> /run/secrets.d/1
drwxr-x--x  3 root keys    0 Nov 11 19:42 secrets.d
lrwxrwxrwx  1 root root   26 Nov 11 19:42 secrets-for-users -> /run/secrets-for-users.d/1
drwxr-x--x  3 root root    0 Nov 11 19:42 secrets-for-users.d
...

This looks reasonable to me: secrets is a symlink, as expected.

[DrymarchonShaun]

@jfly what if you set the path of example_key, something like

diff --git a/nix-sops.nix b/nix-sops.nix
index 5fb72ce..31c9b9f 100644
--- a/nix-sops.nix
+++ b/nix-sops.nix
@@ -6,7 +6,9 @@
   # disable importing host ssh keys
   sops.gnupg.sshKeyPaths = [];
 
-  sops.secrets.example_key = {};
+  sops.secrets.example_key = {
+    path = "${config.users.users.example.home}/example.txt";
+  };
   sops.templates."some.conf".content = ''
     pass = ${config.sops.placeholder.example_key}
   '';

I'd test it myself but nixos-shell doesn't want to work for me

[neumantm]

Hm thats weird. For me /run/secrets was a not-symlinked folder containing only the folder rendered.
I will investigate later.