access forbidden by rule

[clayrisser]

The WordPress pod crashes because the readiness probe fails to ping the nginx server. I get the following error.

Sun, Dec 13 2020 9:11:07 pm | 2020/12/14 03:11:07 [error] 52#52: *93 access forbidden by rule, client: 178.128.11.1, server: tmp-wordpress-wordpress-5f75494c76-gcx65, request: "GET /-/php-ping HTTP/1.1", host: "10.42.1.159:8080"
Sun, Dec 13 2020 9:11:12 pm | 2020/12/14 03:11:12 [error] 52#52: *136 access forbidden by rule, client: 178.128.11.1, server: tmp-wordpress-wordpress-5f75494c76-gcx65, request: "GET /-/php-ping HTTP/1.1", host: "10.42.1.159:8080"
Sun, Dec 13 2020 9:11:17 pm | 2020/12/14 03:11:17 [error] 52#52: *179 access forbidden by rule, client: 178.128.11.1, server: tmp-wordpress-wordpress-5f75494c76-gcx65, request: "GET /-/php-ping HTTP/1.1", host: "10.42.1.159:8080"

Expect

WordPress to run without crashing.

Reproduce

Create WordPress CRD on kubernetes using wordpress operator.

Below is the full log.

time="2020-12-14T03:10:55Z" level=info msg="create process:php-fpm"
--
Sun, Dec 13 2020 9:10:55 pm | time="2020-12-14T03:10:55Z" level=info msg="create process:nginx"
Sun, Dec 13 2020 9:10:55 pm | time="2020-12-14T03:10:55Z" level=info msg="stop listening"
Sun, Dec 13 2020 9:10:55 pm | time="2020-12-14T03:10:55Z" level=info msg="try to start program" program=php-fpm
Sun, Dec 13 2020 9:10:55 pm | time="2020-12-14T03:10:55Z" level=info msg="try to start program" program=nginx
Sun, Dec 13 2020 9:10:55 pm | time="2020-12-14T03:10:55Z" level=info msg="success to listen on address" addr="127.0.0.1:9001" protocol=tcp
Sun, Dec 13 2020 9:10:55 pm | time="2020-12-14T03:10:55Z" level=debug msg="wait program exit" program=php-fpm
Sun, Dec 13 2020 9:10:55 pm | time="2020-12-14T03:10:55Z" level=debug msg="wait program exit" program=nginx
Sun, Dec 13 2020 9:10:55 pm | [start-nginx] Initiated start-nginx.sh script.
Sun, Dec 13 2020 9:10:55 pm | 2020/12/14 03:10:55 Waiting for: unix:///var/run/php-www.sock
Sun, Dec 13 2020 9:10:55 pm | 2020/12/14 03:10:55 Problem with dial: dial unix /var/run/php-www.sock: connect: no such file or directory. Sleeping 1s
Sun, Dec 13 2020 9:10:55 pm | [14-Dec-2020 03:10:55] NOTICE: fpm is running, pid 29
Sun, Dec 13 2020 9:10:55 pm | [14-Dec-2020 03:10:55] NOTICE: ready to handle connections
Sun, Dec 13 2020 9:10:56 pm | time="2020-12-14T03:10:56Z" level=info msg="success to start program" program=php-fpm
Sun, Dec 13 2020 9:10:56 pm | time="2020-12-14T03:10:56Z" level=info msg="success to start program" program=nginx
Sun, Dec 13 2020 9:10:56 pm | 2020/12/14 03:10:56 Connected to unix:///var/run/php-www.sock
Sun, Dec 13 2020 9:10:56 pm | 2020/12/14 03:10:56 [notice] 48#48: using the "epoll" event method
Sun, Dec 13 2020 9:10:56 pm | 2020/12/14 03:10:56 [notice] 48#48: openresty/1.13.6.2
Sun, Dec 13 2020 9:10:56 pm | 2020/12/14 03:10:56 [notice] 48#48: OS: Linux 5.4.0-51-generic
Sun, Dec 13 2020 9:10:56 pm | 2020/12/14 03:10:56 [notice] 48#48: getrlimit(RLIMIT_NOFILE): 1048576:1048576
Sun, Dec 13 2020 9:10:56 pm | 2020/12/14 03:10:56 [notice] 48#48: start worker processes
Sun, Dec 13 2020 9:10:56 pm | 2020/12/14 03:10:56 [notice] 48#48: start worker process 49
Sun, Dec 13 2020 9:10:56 pm | 2020/12/14 03:10:56 [notice] 48#48: start worker process 50
Sun, Dec 13 2020 9:10:56 pm | 2020/12/14 03:10:56 [notice] 48#48: start worker process 51
Sun, Dec 13 2020 9:10:56 pm | 2020/12/14 03:10:56 [notice] 48#48: start worker process 52
Sun, Dec 13 2020 9:11:07 pm | 2020/12/14 03:11:07 [error] 52#52: *93 access forbidden by rule, client: 178.128.11.1, server: tmp-wordpress-wordpress-5f75494c76-gcx65, request: "GET /-/php-ping HTTP/1.1", host: "10.42.1.159:8080"
Sun, Dec 13 2020 9:11:12 pm | 2020/12/14 03:11:12 [error] 52#52: *136 access forbidden by rule, client: 178.128.11.1, server: tmp-wordpress-wordpress-5f75494c76-gcx65, request: "GET /-/php-ping HTTP/1.1", host: "10.42.1.159:8080"
Sun, Dec 13 2020 9:11:17 pm | 2020/12/14 03:11:17 [error] 52#52: *179 access forbidden by rule, client: 178.128.11.1, server: tmp-wordpress-wordpress-5f75494c76-gcx65, request: "GET /-/php-ping HTTP/1.1", host: "10.42.1.159:8080"
Sun, Dec 13 2020 9:11:18 pm | [14-Dec-2020 03:11:18] NOTICE: [pool www-async] child 44 exited with code 0 after 22.602159 seconds from start
Sun, Dec 13 2020 9:11:18 pm | time="2020-12-14T03:11:18Z" level=info msg="receive a signal to stop all process & exit" signal=terminated
Sun, Dec 13 2020 9:11:18 pm | time="2020-12-14T03:11:18Z" level=info msg="stop the program" program=nginx
Sun, Dec 13 2020 9:11:18 pm | time="2020-12-14T03:11:18Z" level=info msg="send stop signal to program" program=nginx signal=QUIT
Sun, Dec 13 2020 9:11:18 pm | time="2020-12-14T03:11:18Z" level=info msg="stop the program" program=php-fpm
Sun, Dec 13 2020 9:11:18 pm | [start-nginx] Stopping nginx.
Sun, Dec 13 2020 9:11:18 pm | [start-nginx] Killing subprocesses of process 28.
Sun, Dec 13 2020 9:11:18 pm | time="2020-12-14T03:11:18Z" level=info msg="send stop signal to program" program=php-fpm signal=QUIT
Sun, Dec 13 2020 9:11:18 pm | [start-php-fpm] Waiting for nginx to stop.
Sun, Dec 13 2020 9:11:18 pm | [14-Dec-2020 03:11:18] NOTICE: [pool www-async] child 59 started
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 48#48: signal 3 (SIGQUIT) received from 62, shutting down
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 52#52: gracefully shutting down
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 51#51: gracefully shutting down
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 51#51: exiting
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 52#52: exiting
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 52#52: exit
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 51#51: exit
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 50#50: gracefully shutting down
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 50#50: exiting
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 48#48: signal 17 (SIGCHLD) received from 52
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 48#48: worker process 52 exited with code 0
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 48#48: signal 29 (SIGIO) received
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 50#50: exit
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 48#48: signal 17 (SIGCHLD) received from 50
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 48#48: worker process 50 exited with code 0
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 48#48: signal 29 (SIGIO) received
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 48#48: signal 17 (SIGCHLD) received from 51
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 49#49: gracefully shutting down
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 48#48: worker process 51 exited with code 0
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 48#48: signal 29 (SIGIO) received
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 49#49: exiting
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 49#49: exit
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 48#48: signal 17 (SIGCHLD) received from 49
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 48#48: worker process 49 exited with code 0
Sun, Dec 13 2020 9:11:18 pm | 2020/12/14 03:11:18 [notice] 48#48: exit
Sun, Dec 13 2020 9:11:18 pm | [start-nginx] Stopped nginx.
Sun, Dec 13 2020 9:11:18 pm | [start-php-fpm] Stopping php-fpm.
Sun, Dec 13 2020 9:11:18 pm | [14-Dec-2020 03:11:18] NOTICE: Finishing ...
Sun, Dec 13 2020 9:11:18 pm | [14-Dec-2020 03:11:18] NOTICE: exiting, bye-bye!
Sun, Dec 13 2020 9:11:18 pm | time="2020-12-14T03:11:18Z" level=info msg="program stopped with status:exit status 0" program=nginx
Sun, Dec 13 2020 9:11:18 pm | time="2020-12-14T03:11:18Z" level=info msg="program exited" program=nginx
Sun, Dec 13 2020 9:11:18 pm | time="2020-12-14T03:11:18Z" level=info msg="Stopped by user, don't start it again" program=nginx
Sun, Dec 13 2020 9:11:18 pm | [start-php-fpm] Stopped php-fpm.
Sun, Dec 13 2020 9:11:18 pm | time="2020-12-14T03:11:18Z" level=info msg="program stopped with status:exit status 0" program=php-fpm
Sun, Dec 13 2020 9:11:18 pm | time="2020-12-14T03:11:18Z" level=info msg="program exited" program=php-fpm
Sun, Dec 13 2020 9:11:18 pm | time="2020-12-14T03:11:18Z" level=info msg="Stopped by user, don't start it again" program=php-fpm

[talss89]

We've experienced this issue too, and the quick answer is that this is a configuration issue with your K8s cluster.

Looking at your logs, the liveness probe URL is being accessed from your WAN interface on a node.

The access control rules for the liveness probe endpoint in this wordpress-runtime image reject access from outside the cluster private subnet of 10.0.0.0/8. You can override this CIDR by setting the STATS_WHITELIST_CIDR env variable (it appears). Here's where this gets applied to the nginx configuration:

stack-runtimes/php/docker/templates/nginx-vhost-conf.d/30-ping.conf

Lines 16 to 18 in 3e93cd0

	{{- if (default "10.0.0.0/8" .Env.STATS_WHITELIST_CIDR) }}
	allow {{ default "10.0.0.0/8" .Env.STATS_WHITELIST_CIDR }};
	{{- end }}

We're waiting to hear back from our service provider (Linode / LKE) to see what can be done at their end. I'd suggest doing the same, or looking at your routing table on nodes.

Suggest this issue gets closed, as it's not a bug in bitpoke/stack-runtimes.