404, containers not found when doing `kayobe overcloud container image pull`

[kees-closed]

When running kayobe overcloud container image pull I get the errors below. It looks like it cannot get the containers, e.g. Not Found (\"manifest for 192.168.33.5:4000/openstack.kolla/kolla-toolbox:2023.1-rocky-9 not found: manifest unknown: manifest unknown\

I'm using Rocky 9.

TASK [service-images-pull : common | Pull images] *********************************************************************************************************************************************************************************************
FAILED - RETRYING: [controller0]: common | Pull images (3 retries left).
FAILED - RETRYING: [compute0]: common | Pull images (3 retries left).
FAILED - RETRYING: [controller0]: common | Pull images (2 retries left).
FAILED - RETRYING: [compute0]: common | Pull images (2 retries left).
FAILED - RETRYING: [controller0]: common | Pull images (1 retries left).
FAILED - RETRYING: [compute0]: common | Pull images (1 retries left).
failed: [controller0] (item=fluentd) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "item": {"key": "fluentd", "value": {"container_name": "fluentd", "dimensions": {"ulimits": {"nofile": {"hard": 1048576, "soft": 1048576}}}, "enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS"}, "group": "fluentd", "image": "192.168.33.5:4000/openstack.kolla/fluentd:2023.1-rocky-9", "volumes": ["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "", "kolla_logs:/var/log/kolla/", "fluentd_data:/var/lib/fluentd/data/"]}}, "msg": "'Traceback (most recent call last):\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/client.py\", line 268, in _raise_for_status\\n    response.raise_for_status()\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/requests/models.py\", line 1021, in raise_for_status\\n    raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.46/images/create?tag=2023.1-rocky-9&fromImage=192.168.33.5%3A4000%2Fopenstack.kolla%2Ffluentd\\n\\nThe above exception was the direct cause of the following exception:\\n\\nTraceback (most recent call last):\\n  File \"/tmp/ansible_kolla_docker_payload_g12zns26/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 408, in main\\n  File \"/tmp/ansible_kolla_docker_payload_g12zns26/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 211, in pull_image\\n    json.loads(line.strip().decode(\\'utf-8\\')) for line in self.dc.pull(\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/image.py\", line 429, in pull\\n    self._raise_for_status(response)\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/client.py\", line 270, in _raise_for_status\\n    raise create_api_error_from_http_exception(e) from e\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/errors.py\", line 39, in create_api_error_from_http_exception\\n    raise cls(e, response=response, explanation=explanation) from e\\ndocker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.46/images/create?tag=2023.1-rocky-9&fromImage=192.168.33.5%3A4000%2Fopenstack.kolla%2Ffluentd: Not Found (\"manifest for 192.168.33.5:4000/openstack.kolla/fluentd:2023.1-rocky-9 not found: manifest unknown: manifest unknown\")\\n'"}
failed: [compute0] (item=fluentd) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "item": {"key": "fluentd", "value": {"container_name": "fluentd", "dimensions": {"ulimits": {"nofile": {"hard": 1048576, "soft": 1048576}}}, "enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS"}, "group": "fluentd", "image": "192.168.33.5:4000/openstack.kolla/fluentd:2023.1-rocky-9", "volumes": ["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "", "kolla_logs:/var/log/kolla/", "fluentd_data:/var/lib/fluentd/data/"]}}, "msg": "'Traceback (most recent call last):\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/client.py\", line 268, in _raise_for_status\\n    response.raise_for_status()\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/requests/models.py\", line 1021, in raise_for_status\\n    raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.46/images/create?tag=2023.1-rocky-9&fromImage=192.168.33.5%3A4000%2Fopenstack.kolla%2Ffluentd\\n\\nThe above exception was the direct cause of the following exception:\\n\\nTraceback (most recent call last):\\n  File \"/tmp/ansible_kolla_docker_payload_395v2eh2/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 408, in main\\n  File \"/tmp/ansible_kolla_docker_payload_395v2eh2/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 211, in pull_image\\n    json.loads(line.strip().decode(\\'utf-8\\')) for line in self.dc.pull(\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/image.py\", line 429, in pull\\n    self._raise_for_status(response)\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/client.py\", line 270, in _raise_for_status\\n    raise create_api_error_from_http_exception(e) from e\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/errors.py\", line 39, in create_api_error_from_http_exception\\n    raise cls(e, response=response, explanation=explanation) from e\\ndocker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.46/images/create?tag=2023.1-rocky-9&fromImage=192.168.33.5%3A4000%2Fopenstack.kolla%2Ffluentd: Not Found (\"manifest for 192.168.33.5:4000/openstack.kolla/fluentd:2023.1-rocky-9 not found: manifest unknown: manifest unknown\")\\n'"}
FAILED - RETRYING: [controller0]: common | Pull images (3 retries left).
FAILED - RETRYING: [compute0]: common | Pull images (3 retries left).
FAILED - RETRYING: [controller0]: common | Pull images (2 retries left).
FAILED - RETRYING: [compute0]: common | Pull images (2 retries left).
FAILED - RETRYING: [controller0]: common | Pull images (1 retries left).
FAILED - RETRYING: [compute0]: common | Pull images (1 retries left).
failed: [compute0] (item=kolla-toolbox) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "item": {"key": "kolla-toolbox", "value": {"container_name": "kolla_toolbox", "dimensions": {"ulimits": {"nofile": {"hard": 1048576, "soft": 1048576}}}, "enabled": true, "environment": {"ANSIBLE_LIBRARY": "/usr/share/ansible", "ANSIBLE_NOCOLOR": "1"}, "group": "kolla-toolbox", "image": "192.168.33.5:4000/openstack.kolla/kolla-toolbox:2023.1-rocky-9", "privileged": true, "volumes": ["/etc/kolla/kolla-toolbox/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "", "/dev/:/dev/", "/run/:/run/:shared", "kolla_logs:/var/log/kolla/"]}}, "msg": "'Traceback (most recent call last):\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/client.py\", line 268, in _raise_for_status\\n    response.raise_for_status()\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/requests/models.py\", line 1021, in raise_for_status\\n    raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.46/images/create?tag=2023.1-rocky-9&fromImage=192.168.33.5%3A4000%2Fopenstack.kolla%2Fkolla-toolbox\\n\\nThe above exception was the direct cause of the following exception:\\n\\nTraceback (most recent call last):\\n  File \"/tmp/ansible_kolla_docker_payload_njr98kxy/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 408, in main\\n  File \"/tmp/ansible_kolla_docker_payload_njr98kxy/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 211, in pull_image\\n    json.loads(line.strip().decode(\\'utf-8\\')) for line in self.dc.pull(\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/image.py\", line 429, in pull\\n    self._raise_for_status(response)\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/client.py\", line 270, in _raise_for_status\\n    raise create_api_error_from_http_exception(e) from e\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/errors.py\", line 39, in create_api_error_from_http_exception\\n    raise cls(e, response=response, explanation=explanation) from e\\ndocker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.46/images/create?tag=2023.1-rocky-9&fromImage=192.168.33.5%3A4000%2Fopenstack.kolla%2Fkolla-toolbox: Not Found (\"manifest for 192.168.33.5:4000/openstack.kolla/kolla-toolbox:2023.1-rocky-9 not found: manifest unknown: manifest unknown\")\\n'"}
failed: [controller0] (item=kolla-toolbox) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "item": {"key": "kolla-toolbox", "value": {"container_name": "kolla_toolbox", "dimensions": {"ulimits": {"nofile": {"hard": 1048576, "soft": 1048576}}}, "enabled": true, "environment": {"ANSIBLE_LIBRARY": "/usr/share/ansible", "ANSIBLE_NOCOLOR": "1"}, "group": "kolla-toolbox", "image": "192.168.33.5:4000/openstack.kolla/kolla-toolbox:2023.1-rocky-9", "privileged": true, "volumes": ["/etc/kolla/kolla-toolbox/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "", "/dev/:/dev/", "/run/:/run/:shared", "kolla_logs:/var/log/kolla/"]}}, "msg": "'Traceback (most recent call last):\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/client.py\", line 268, in _raise_for_status\\n    response.raise_for_status()\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/requests/models.py\", line 1021, in raise_for_status\\n    raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.46/images/create?tag=2023.1-rocky-9&fromImage=192.168.33.5%3A4000%2Fopenstack.kolla%2Fkolla-toolbox\\n\\nThe above exception was the direct cause of the following exception:\\n\\nTraceback (most recent call last):\\n  File \"/tmp/ansible_kolla_docker_payload_mo06syvn/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 408, in main\\n  File \"/tmp/ansible_kolla_docker_payload_mo06syvn/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 211, in pull_image\\n    json.loads(line.strip().decode(\\'utf-8\\')) for line in self.dc.pull(\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/image.py\", line 429, in pull\\n    self._raise_for_status(response)\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/client.py\", line 270, in _raise_for_status\\n    raise create_api_error_from_http_exception(e) from e\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/errors.py\", line 39, in create_api_error_from_http_exception\\n    raise cls(e, response=response, explanation=explanation) from e\\ndocker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.46/images/create?tag=2023.1-rocky-9&fromImage=192.168.33.5%3A4000%2Fopenstack.kolla%2Fkolla-toolbox: Not Found (\"manifest for 192.168.33.5:4000/openstack.kolla/kolla-toolbox:2023.1-rocky-9 not found: manifest unknown: manifest unknown\")\\n'"}
FAILED - RETRYING: [compute0]: common | Pull images (3 retries left).
FAILED - RETRYING: [controller0]: common | Pull images (3 retries left).
FAILED - RETRYING: [compute0]: common | Pull images (2 retries left).
FAILED - RETRYING: [controller0]: common | Pull images (2 retries left).
FAILED - RETRYING: [compute0]: common | Pull images (1 retries left).
FAILED - RETRYING: [controller0]: common | Pull images (1 retries left).
failed: [compute0] (item=cron) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "item": {"key": "cron", "value": {"container_name": "cron", "dimensions": {"ulimits": {"nofile": {"hard": 1048576, "soft": 1048576}}}, "enabled": true, "environment": {"KOLLA_LOGROTATE_SCHEDULE": "daily"}, "group": "cron", "image": "192.168.33.5:4000/openstack.kolla/cron:2023.1-rocky-9", "volumes": ["/etc/kolla/cron/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "", "kolla_logs:/var/log/kolla/"]}}, "msg": "'Traceback (most recent call last):\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/client.py\", line 268, in _raise_for_status\\n    response.raise_for_status()\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/requests/models.py\", line 1021, in raise_for_status\\n    raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.46/images/create?tag=2023.1-rocky-9&fromImage=192.168.33.5%3A4000%2Fopenstack.kolla%2Fcron\\n\\nThe above exception was the direct cause of the following exception:\\n\\nTraceback (most recent call last):\\n  File \"/tmp/ansible_kolla_docker_payload_uvayzahh/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 408, in main\\n  File \"/tmp/ansible_kolla_docker_payload_uvayzahh/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 211, in pull_image\\n    json.loads(line.strip().decode(\\'utf-8\\')) for line in self.dc.pull(\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/image.py\", line 429, in pull\\n    self._raise_for_status(response)\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/client.py\", line 270, in _raise_for_status\\n    raise create_api_error_from_http_exception(e) from e\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/errors.py\", line 39, in create_api_error_from_http_exception\\n    raise cls(e, response=response, explanation=explanation) from e\\ndocker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.46/images/create?tag=2023.1-rocky-9&fromImage=192.168.33.5%3A4000%2Fopenstack.kolla%2Fcron: Not Found (\"manifest for 192.168.33.5:4000/openstack.kolla/cron:2023.1-rocky-9 not found: manifest unknown: manifest unknown\")\\n'"}
failed: [controller0] (item=cron) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "item": {"key": "cron", "value": {"container_name": "cron", "dimensions": {"ulimits": {"nofile": {"hard": 1048576, "soft": 1048576}}}, "enabled": true, "environment": {"KOLLA_LOGROTATE_SCHEDULE": "daily"}, "group": "cron", "image": "192.168.33.5:4000/openstack.kolla/cron:2023.1-rocky-9", "volumes": ["/etc/kolla/cron/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "", "kolla_logs:/var/log/kolla/"]}}, "msg": "'Traceback (most recent call last):\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/client.py\", line 268, in _raise_for_status\\n    response.raise_for_status()\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/requests/models.py\", line 1021, in raise_for_status\\n    raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.46/images/create?tag=2023.1-rocky-9&fromImage=192.168.33.5%3A4000%2Fopenstack.kolla%2Fcron\\n\\nThe above exception was the direct cause of the following exception:\\n\\nTraceback (most recent call last):\\n  File \"/tmp/ansible_kolla_docker_payload_8tkigs6y/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 408, in main\\n  File \"/tmp/ansible_kolla_docker_payload_8tkigs6y/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 211, in pull_image\\n    json.loads(line.strip().decode(\\'utf-8\\')) for line in self.dc.pull(\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/image.py\", line 429, in pull\\n    self._raise_for_status(response)\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/api/client.py\", line 270, in _raise_for_status\\n    raise create_api_error_from_http_exception(e) from e\\n  File \"/opt/kayobe/venvs/kolla-ansible/lib64/python3.9/site-packages/docker/errors.py\", line 39, in create_api_error_from_http_exception\\n    raise cls(e, response=response, explanation=explanation) from e\\ndocker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.46/images/create?tag=2023.1-rocky-9&fromImage=192.168.33.5%3A4000%2Fopenstack.kolla%2Fcron: Not Found (\"manifest for 192.168.33.5:4000/openstack.kolla/cron:2023.1-rocky-9 not found: manifest unknown: manifest unknown\")\\n'"}

PLAY RECAP ************************************************************************************************************************************************************************************************************************************
compute0                   : ok=5    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0   
controller0                : ok=5    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0   

kolla-ansible pull exited 2

The VMs are running:

(kayobe-venv) [rocky@ip-172-31-34-228 kayobe]$ sudo virsh list --all
 Id   Name          State
-----------------------------
 1    seed          running
 2    compute0      running
 3    controller0   running

The port 4000 is available, but the images are not present.

[rocky@seed ~]$ sudo ss -wauntlsp
Total: 203
TCP:   25 (estab 15, closed 0, orphaned 0, timewait 0)

Transport Total     IP        IPv6
RAW       0         0         0        
UDP       7         5         2        
TCP       25        9         16       
INET      32        14        18       
FRAG      0         0         0        

Netid           State            Recv-Q           Send-Q                     Local Address:Port                      Peer Address:Port           Process                                                                                       
udp             UNCONN           0                0                                0.0.0.0:67                             0.0.0.0:*               users:(("dnsmasq",pid=1722,fd=4))                                                            
udp             UNCONN           0                0                           192.168.33.5:69                             0.0.0.0:*               users:(("dnsmasq",pid=1722,fd=6))                                                            
udp             UNCONN           0                0                                0.0.0.0:111                            0.0.0.0:*               users:(("rpcbind",pid=692,fd=5),("systemd",pid=1,fd=33))                                     
udp             UNCONN           0                0                                0.0.0.0:123                            0.0.0.0:*               users:(("chronyd",pid=753,fd=7))                                                             
udp             UNCONN           0                0                              127.0.0.1:323                            0.0.0.0:*               users:(("chronyd",pid=753,fd=5))                                                             
udp             UNCONN           0                0                                   [::]:111                               [::]:*               users:(("rpcbind",pid=692,fd=7),("systemd",pid=1,fd=35))                                     
udp             UNCONN           0                0                                  [::1]:323                               [::]:*               users:(("chronyd",pid=753,fd=6))                                                             
tcp             LISTEN           0                511                              0.0.0.0:5050                           0.0.0.0:*               users:(("nginx",pid=1809,fd=10),("nginx",pid=1807,fd=10),("nginx",pid=1806,fd=10))           
tcp             LISTEN           0                511                              0.0.0.0:6385                           0.0.0.0:*               users:(("nginx",pid=1809,fd=9),("nginx",pid=1807,fd=9),("nginx",pid=1806,fd=9))              
tcp             LISTEN           0                128                              0.0.0.0:22                             0.0.0.0:*               users:(("sshd",pid=1140,fd=3))                                                               
tcp             LISTEN           0                4096                             0.0.0.0:111                            0.0.0.0:*               users:(("rpcbind",pid=692,fd=4),("systemd",pid=1,fd=31))                                     
tcp             LISTEN           0                511                              0.0.0.0:8083                           0.0.0.0:*               users:(("nginx",pid=1809,fd=8),("nginx",pid=1807,fd=8),("nginx",pid=1806,fd=8))              
tcp             LISTEN           0                511                              0.0.0.0:8080                           0.0.0.0:*               users:(("nginx",pid=1809,fd=7),("nginx",pid=1807,fd=7),("nginx",pid=1806,fd=7))              
tcp             LISTEN           0                128                                 [::]:22                                [::]:*               users:(("sshd",pid=1140,fd=4))                                                               
tcp             LISTEN           0                4096                                [::]:111                               [::]:*               users:(("rpcbind",pid=692,fd=6),("systemd",pid=1,fd=34))                                     
tcp             LISTEN           0                4096                                   *:4000                                 *:*               users:(("registry",pid=1600,fd=6))                                                           
tcp             LISTEN           0                80                                     *:3306                                 *:*               users:(("mariadbd",pid=1831,fd=44)) 

I can list some images on the container host though:

[root@seed containerd]# docker images                                                                                                                                                                                                          
REPOSITORY                                                    TAG              IMAGE ID       CREATED             SIZE                                                                                                                         
192.168.33.5:4000/openstack.kolla/horizon                     2023.1-rocky-9   fd061c3ced56   About an hour ago   1.37GB
192.168.33.5:4000/openstack.kolla/nova-compute                2023.1-rocky-9   4b26c658e7c7   2 hours ago         2.01GB
192.168.33.5:4000/openstack.kolla/neutron-server              2023.1-rocky-9   6ac187457dae   2 hours ago         1.38GB
192.168.33.5:4000/openstack.kolla/neutron-l3-agent            2023.1-rocky-9   6ffb39c4567a   2 hours ago         1.41GB
192.168.33.5:4000/openstack.kolla/neutron-linuxbridge-agent   2023.1-rocky-9   ed79bb85713b   2 hours ago         1.38GB
192.168.33.5:4000/openstack.kolla/neutron-metadata-agent      2023.1-rocky-9   005efbd6a651   2 hours ago         1.37GB
192.168.33.5:4000/openstack.kolla/neutron-openvswitch-agent   2023.1-rocky-9   25383d79113b   2 hours ago         1.37GB
192.168.33.5:4000/openstack.kolla/neutron-dhcp-agent          2023.1-rocky-9   4cf4896851ce   2 hours ago         1.37GB
192.168.33.5:4000/openstack.kolla/neutron-base                2023.1-rocky-9   fabfd816df89   2 hours ago         1.37GB
192.168.33.5:4000/openstack.kolla/nova-ssh                    2023.1-rocky-9   da4052c5cd27   2 hours ago         1.46GB
192.168.33.5:4000/openstack.kolla/nova-compute-ironic         2023.1-rocky-9   329487f91e1a   2 hours ago         1.46GB
192.168.33.5:4000/openstack.kolla/nova-novncproxy             2023.1-rocky-9   d713435fbf97   2 hours ago         1.46GB
192.168.33.5:4000/openstack.kolla/nova-conductor              2023.1-rocky-9   084d65064dd0   2 hours ago         1.43GB
192.168.33.5:4000/openstack.kolla/nova-api                    2023.1-rocky-9   3cd3320e06bf   2 hours ago         1.43GB
192.168.33.5:4000/openstack.kolla/nova-serialproxy            2023.1-rocky-9   758271b5437e   2 hours ago         1.43GB
192.168.33.5:4000/openstack.kolla/nova-scheduler              2023.1-rocky-9   83a783c028eb   2 hours ago         1.43GB
192.168.33.5:4000/openstack.kolla/nova-base                   2023.1-rocky-9   cee0beffb243   2 hours ago         1.43GB
192.168.33.5:4000/openstack.kolla/keystone-ssh                2023.1-rocky-9   38a0beec984e   3 hours ago         1.23GB
192.168.33.5:4000/openstack.kolla/keystone                    2023.1-rocky-9   c613bb3f4b0b   3 hours ago         1.23GB
192.168.33.5:4000/openstack.kolla/keystone-fernet             2023.1-rocky-9   b4d422d20d9a   3 hours ago         1.23GB
192.168.33.5:4000/openstack.kolla/glance-api                  2023.1-rocky-9   bead5a5bf22c   3 hours ago         1.32GB
192.168.33.5:4000/openstack.kolla/heat-engine                 2023.1-rocky-9   8ba7906ce93b   3 hours ago         1.2GB
192.168.33.5:4000/openstack.kolla/heat-api-cfn                2023.1-rocky-9   41d9fac7670f   3 hours ago         1.2GB                                                                                                                        
192.168.33.5:4000/openstack.kolla/heat-api                    2023.1-rocky-9   4b4ce6421fa0   3 hours ago         1.2GB 
192.168.33.5:4000/openstack.kolla/keystone-base               2023.1-rocky-9   077bd504d5ef   3 hours ago         1.2GB
192.168.33.5:4000/openstack.kolla/heat-base                   2023.1-rocky-9   0369544a335e   3 hours ago         1.2GB
192.168.33.5:4000/openstack.kolla/glance-base                 2023.1-rocky-9   9a793a81d0b6   3 hours ago         1.28GB
192.168.33.5:4000/openstack.kolla/placement-api               2023.1-rocky-9   80f920e05a91   3 hours ago         1.14GB
192.168.33.5:4000/openstack.kolla/placement-base              2023.1-rocky-9   2e36952399fd   3 hours ago         1.14GB
192.168.33.5:4000/openstack.kolla/openstack-base              2023.1-rocky-9   cf8620ad539c   4 hours ago         1.13GB
192.168.33.5:4000/openstack.kolla/kolla-toolbox               2023.1-rocky-9   f27cf4a96eaa   4 hours ago         1.08GB
192.168.33.5:4000/openstack.kolla/nova-libvirt                2023.1-rocky-9   dc5db27f2c5d   4 hours ago         1.4GB 
192.168.33.5:4000/openstack.kolla/mariadb-server              2023.1-rocky-9   7addc216b29f   4 hours ago         1.02GB
192.168.33.5:4000/openstack.kolla/openvswitch-netcontrold     2023.1-rocky-9   a7fd36aa4823   4 hours ago         716MB                                                                                                                        
192.168.33.5:4000/openstack.kolla/fluentd                     2023.1-rocky-9   5653e864d7cc   5 hours ago         854MB                                                                                                                        
192.168.33.5:4000/openstack.kolla/openvswitch-vswitchd        2023.1-rocky-9   e24583edb5dd   5 hours ago         696MB                                                                                                                        
192.168.33.5:4000/openstack.kolla/rabbitmq-3-13               2023.1-rocky-9   e5320271d9f2   5 hours ago         679MB                                                                                                                        
192.168.33.5:4000/openstack.kolla/openvswitch-db-server       2023.1-rocky-9   cdfb0cdbad18   5 hours ago         696MB                                                                                                                        
192.168.33.5:4000/openstack.kolla/openvswitch-base            2023.1-rocky-9   60b26b3523f8   5 hours ago         696MB                                                                                                                        
192.168.33.5:4000/openstack.kolla/mariadb-clustercheck        2023.1-rocky-9   1636e8abbeae   5 hours ago         722MB 
192.168.33.5:4000/openstack.kolla/rabbitmq-3-12               2023.1-rocky-9   82a873f19fcf   5 hours ago         678MB                                                                                                                        
192.168.33.5:4000/openstack.kolla/rabbitmq                    2023.1-rocky-9   9af52d9d6126   5 hours ago         677MB                                                                                                                        
192.168.33.5:4000/openstack.kolla/mariadb-base                2023.1-rocky-9   3ff2323f2908   5 hours ago         722MB                                                                                                                        
192.168.33.5:4000/openstack.kolla/keepalived                  2023.1-rocky-9   8615c3f3b8e4   5 hours ago         647MB
192.168.33.5:4000/openstack.kolla/memcached                   2023.1-rocky-9   1bbc00431e49   5 hours ago         639MB
192.168.33.5:4000/openstack.kolla/haproxy                     2023.1-rocky-9   d4cee1385f09   5 hours ago         626MB
192.168.33.5:4000/openstack.kolla/cron                        2023.1-rocky-9   24c99a446187   5 hours ago         611MB                                                                                                                        
192.168.33.5:4000/openstack.kolla/base                        2023.1-rocky-9   52e426b63b66   6 hours ago         598MB                                                                                                                        
192.168.33.5:4000/openstack.kolla/bifrost-deploy              2023.1-rocky-9   1418bce8e37c   12 days ago         2.15GB

What am I missing? I followed the steps in sequence. I also re-ran them. But this issue keeps happening.

[kees-closed]

I guess this is not a common issue? I tried to re-run the steps already, but I get the same result.

[rplazevedo]

I guess this is not a common issue? I tried to re-run the steps already, but I get the same result.

@kees-closed I am having thee same issue. Were you ever able to solve it?

[priteau]

Hello @kees-closed and @rplazevedo. These containers should have been pushed to the Docker registry by the command ./config/src/kayobe-config/pull-retag-push-images.sh. Can you try running this script again? Is there any error?

[rplazevedo]

@priteau Yeah, there is an error during the push:

TASK [Push container images (may take a long time)] **********************************************************************************************************************************************************************************
failed: [seed] (item=bifrost-deploy) => {"ansible_loop_var": "item", "changed": true, "cmd": ["docker", "push", "192.168.33.5:4000/openstack.kolla/bifrost-deploy:2023.1-rocky-9"], "delta": "0:00:00.031763", "end": "2024-10-30 14:31:44.811053", "item": "bifrost-deploy", "msg": "non-zero return code", "rc": 1, "start": "2024-10-30 14:31:44.779290", "stderr": "Get \"http://192.168.33.5:4000/v2/\": dial tcp 192.168.33.5:4000: connect: connection refused", "stderr_lines": ["Get \"http://192.168.33.5:4000/v2/\": dial tcp 192.168.33.5:4000: connect: connection refused"], "stdout": "The push refers to repository [192.168.33.5:4000/openstack.kolla/bifrost-deploy]", "stdout_lines": ["The push refers to repository [192.168.33.5:4000/openstack.kolla/bifrost-deploy]"]}

There was also an error during some of the steps trying to SSH into the host, but those were all ignored, I think. Don't know if they are related.

[rplazevedo]

So I think I fixed it. The docker registry in the seed was not being setup, because while the docker_registry_enabled variable was set to true in ansible/roles/docker-registry/defaults/main.yml, it somehow changed when it went into ansible/roles/docker-registry/tasks/deploy.yml. I edited the latter and hardcoded state: "started", which finally created the registry.