Issues in customizing component paths in kube-bench/cfg/config.yaml

[ashishgavali10]

Hello Team,

We have deployed Kubernetes cluster, due to security reasons we have stored our component manifests in different location.
I have changed the paths in kube-bench/cfg/config.yaml according to our requirement, but the checks are still failing.
Can anyone guide me how to change the paths for manifest file? do we need to make more changes any of the other files?

[lizrice]

As a first step I'd recommend running with -v 3 --logtostderr to get log output that shows where kube-bench checks for different file locations - that should help you check whether the changes to kube-bench/cfg/config.yaml are being picked up correctly.

Note that by default, if you're running kube-bench from a container image, the config files are included in that image, so you would need to either

• rebuild the image to include the new version of that config file
• mount a config directory with the updated config file into the container.

[ashishgavali10]

Hello Liz,

I modified path for kubeapiserver as below in kube-bench/cfg/config.yaml :
defaultconf: /var/vcap/jobs/kube-apiserver/config/bpm.yml

Executed kube bench with below command:
./kube-bench master -v 3 --logtostderr

As newly updated paths are being picked while scan but still checks are failing even though file permissions and users are configured correctly on manifest file. Do I need to make changes in any of the configuration file or am I missing anything?

Adding output snippet for better understanding:

I0911 10:43:52.852963 170631 util.go:195] Using default config file name '/var/vcap/jobs/kube-apiserver/config/bpm.yml' for component apiserver

[lizrice]

We can see from the logs that the filename is being successfully substituted, but there's no output from the test command, which suggests that the file doesn't exist. What happens if you run the test directly, something like this?

$ export apiserverconf="./cfg/config.yaml"
$ /bin/sh -c 'if test -e $apiserverconf; then stat -c permissions=%a $apiserverconf; fi'
permissions=644

In that example I've just used a file that I know exists, the cfg/config.yaml file. If the file doesn't exist you'll get empty output (like what we see in the logs)

$ export apiserverconf="not real"
$ /bin/sh -c 'if test -e $apiserverconf; then stat -c permissions=%a $apiserverconf; fi'

[ashishgavali10]

Hello Liz,
Issue is resolved now, there was a typo in path. I am able to get expected output with newly updated paths.
I have been asked to execute Kube-bench inside a pod as a Job with the updated paths.

I have done following changes:

1. Updated paths in /cfg/config.yaml file for all the components.
2. Built new image with the docker file present in kube-bench repo, which is taking newly updated configuration file and image is uploaded to docker hub.
3. Edited container/images section job.yml file to fetch newly created docker image.

Job is getting created but pods are failing.

Below are the screenshots of kubectl describe pods command, newly updated images is being used while creating pods, but container failes with exit status code 1.

It is working fine with stock kube-bench image but giving issues with my custom image.
Could you please help me with this.

[mozillazg]

@ashishgavali10 It looks like you are running master related checks on worker node. Your job spec should like https://github.com/aquasecurity/kube-bench/blob/master/job-master.yaml if you want it running on master node or running on worker node with https://github.com/aquasecurity/kube-bench/blob/master/job-node.yaml