Skip to content

Commit 427fb58

Browse files
priteaupriteau
committed
Add custom playbooks to manage Dell firmware
1 parent fc278ab commit 427fb58

File tree

5 files changed

+262
-0
lines changed

5 files changed

+262
-0
lines changed

‎doc/source/operations/dell-firmware-update.rst

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
===============================
2+
Dell Firmware Update Automation
3+
===============================
4+
5+
Overview
6+
========
7+
8+
Custom playbooks are available to automate firmware updates on Dell hardware.
9+
10+
We make use of `Dell Repository Manager (DRM)
11+
<https://www.dell.com/support/kbdoc/en-uk/000177083/support-for-dell-emc-repository-manager-drm>`__.
12+
13+
Prerequisites
14+
=============
15+
16+
DRM needs to listen on port 443 and needs access to the out-of-band management
17+
network. Choose a host where it won't conflict with another service.
18+
19+
To run DRM in a container, first start a container, that has a Docker volume to
20+
host all the firmware files:
21+
22+
.. code-block:: bash
23+
24+
docker volume create dell_firmware
25+
docker run --detach -v dell_firmware:/dell_firmware --name dell-drm --network host --restart always rockylinux:9.3 sleep infinity
26+
27+
Copy in, and then run the installer:
28+
29+
.. code-block:: bash
30+
31+
curl -O https://dl.dell.com/FOLDER11468378M/1/DRMInstaller_3.4.5.938.bin -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0'
32+
docker cp DRMInstaller_3.4.5.938.bin dell-drm:/root
33+
docker exec -it dell-drm bash
34+
cd /root
35+
chmod +x DRMInstaller_3.4.5.938.bin
36+
./DRMInstaller_3.4.5.938.bin
37+
38+
Now you can run DRM, and download a new repo (customise the argument to
39+
``--inputplatformlist`` depending on the targeted hardware):
40+
41+
.. code-block:: bash
42+
43+
/opt/dell/dellrepositorymanager/DRM_Service.sh &
44+
drm --create -r=idrac_repo --inputplatformlist=R640,R6525
45+
drm --deployment-type=share --location=/dell_firmware -r=idrac_repo
46+
47+
Note: sometimes the create call had to be run multiple times before it worked,
48+
with errors relating to ``Unknown platform: R6525``. Restarting the service
49+
might be required.
50+
51+
Now we have the all the files in the Docker volume, we can start Apache to
52+
expose the repo. Use this Dockerfile to support TLS:
53+
54+
.. code-block:: dockerfile
55+
56+
FROM httpd:2.4
57+
58+
RUN sed -i \
59+
-e 's/^#\(Include .*httpd-ssl.conf\)/\1/' \
60+
-e 's/^#\(LoadModule .*mod_ssl.so\)/\1/' \
61+
-e 's/^#\(LoadModule .*mod_socache_shmcb.so\)/\1/' \
62+
-e 's/Listen 80/#Listen 80/' \
63+
conf/httpd.conf
64+
65+
Build a Docker image:
66+
67+
.. code-block:: bash
68+
69+
docker build --network host -t httpd:local .
70+
71+
Generate a self-signed cert:
72+
73+
.. code-block:: bash
74+
75+
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout apache.key -out apache.crt
76+
77+
Run the container:
78+
79+
.. code-block:: bash
80+
81+
docker run -d --name dell-drm-web --network host -v dell_firmware:/usr/local/apache2/htdocs/ -v $PWD/apache.crt:/usr/local/apache2/conf/server.crt -v $PWD/apache.key:/usr/local/apache2/conf/server.key docker.io/library/httpd:local
82+
83+
.. note::
84+
85+
At this point the repository may contain only old version of the firmwares.
86+
Run an update once to make sure the latest files are available (see next
87+
section).
88+
89+
Updating the Repo
90+
=================
91+
92+
At a later date we will want to re-baseline to a new version. The repo
93+
can be updated:
94+
95+
.. code-block:: bash
96+
97+
docker exec -it dell-drm bash
98+
[root@seed /]# drm --update -r=idrac_repo
99+
# check that it has iterated to a new version
100+
[root@seed /]# drm -li=rep
101+
102+
Listing Repositories...
103+
104+
105+
Name Latest version Size Last modified date
106+
---- -------------- ---- -------------
107+
idrac_repo 1.01 4.82 GB 1/9/24 2:22 P.M
108+
109+
# share the new version
110+
[root@seed /]# drm --deployment-type=share --location=/dell_firmware -r=idrac_repo:1.01
111+
[root@seed /]# ls -ltra /dell_firmware | tail -1
112+
-rw-r--r-- 1 root root 7103842 Jan 9 14:24 idrac_repo_Catalog.xml
113+
114+
Then update the ``dell_drm_repo`` variable in ``drac-firmware-update.yml`` if
115+
required.
116+
117+
Manually adding and update file
118+
===============================
119+
120+
Clone an update package the windows format (the iDRAC knows how to process these):
121+
122+
.. code-block:: bash
123+
124+
curl 'https://dl.dell.com/FOLDER09614074M/2/Network_Firmware_77R8T_WN64_22.36.10.10.EXE?uid=39eab3c7-5ad6-4bfc-be6e-b9d09374accd&fn=Network_Firmware_77R8T_WN64_22.36.10.10.EXE' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0' -O Network_Firmware_77R8T_WN64_22.36.10.10.EXE
125+
126+
Import it into your repo:
127+
128+
.. code-block:: bash
129+
130+
drm --import --repository=idrac_repo:1.00 --source=/root --update-package="*.EXE"
131+
132+
Export the repository:
133+
134+
.. code-block:: bash
135+
136+
drm --deployment-type=share --location=/dell_firmware -r=idrac_repo:1.02
137+
138+
Updating firmware versions on a Dell node
139+
=========================================
140+
141+
The updated firmware versions can be applied to a Dell node using the
142+
``drac-firmware-update.yml`` playbook.
143+
144+
The following command will show the list of firmware updates to be applied:
145+
146+
.. code-block:: bash
147+
148+
kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/drac-firmware-update.yml --limit <host>
149+
150+
The following command will apply firmware updates:
151+
152+
.. code-block:: bash
153+
154+
kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/drac-firmware-update.yml --limit <host> -e dell_drm_apply_update=true
155+
156+
.. note::
157+
158+
The playbook will likely fail with an error if the iDRAC firmware is being
159+
updated, since this involves rebooting the iDRAC. Wait for the iDRAC to be
160+
up and run the playbook again to ensure all firmwares have been updated
161+
correctly.

‎doc/source/operations/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ This guide is for operators of the StackHPC Kayobe configuration project.
77
.. toctree::
88
:maxdepth: 1
99

10+
dell-firmware-update
1011
hotfix-playbook
1112
nova-compute-ironic
1213
octavia

‎etc/kayobe/ansible/drac-firmware-inventory.yml

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
---
2+
- hosts: overcloud
3+
gather_facts: false
4+
connection: local
5+
6+
collections:
7+
- dellemc.openmanage
8+
9+
vars:
10+
secrets_ipmi_username: "{{ ipmi_admin_user }}"
11+
secrets_ipmi_password: "{{ ipmi_admin_password }}"
12+
drac_export_path: "{{ playbook_dir }}/idrac_firmware_inventory/"
13+
ansible_python_interpreter: "{{ ansible_playbook_python }}"
14+
tasks:
15+
- name: Get Firmware inventory
16+
idrac_firmware_info:
17+
idrac_ip: "{{ ipmi_address }}"
18+
idrac_user: "{{ secrets_ipmi_username }}"
19+
idrac_password: "{{ secrets_ipmi_password }}"
20+
register: idrac_firmware_output
21+
delegate_to: localhost
22+
when: ipmi_address is defined
23+
24+
- name: Ensure output dir exists
25+
file:
26+
state: directory
27+
path: "{{ drac_export_path }}"
28+
delegate_to: localhost
29+
run_once: True
30+
31+
- name: Write inventory to file
32+
copy:
33+
content: "{{ idrac_firmware_output.firmware_info }}"
34+
dest: "{{ drac_export_path }}/{{ inventory_hostname }}"

‎etc/kayobe/ansible/drac-firmware-update.yml

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
---
2+
- hosts: overcloud
3+
gather_facts: false
4+
connection: local
5+
serial: "{{ lookup('env', 'ANSIBLE_SERIAL') | default(1, true) }}"
6+
7+
collections:
8+
- dellemc.openmanage
9+
10+
vars:
11+
# Run with ``-e dell_drm_apply_update=true`` to apply the firmware updates.
12+
dell_drm_apply_update: False
13+
dell_drm_address: "https://{{ oob_oc_net_name | net_ip(inventory_hostname=groups['seed'][0]) }}"
14+
secrets_ipmi_username: "{{ ipmi_admin_user }}"
15+
secrets_ipmi_password: "{{ ipmi_admin_password }}"
16+
# Look at the README for more details, but you need a repo
17+
# created in drm, and exported via a webserver:
18+
# drm --create -r=idrac_repo --inputplatformlist=R640,R6525
19+
dell_drm_repo: "idrac_repo_Catalog.xml"
20+
# Run with -e bifrost_maintenance=false if nodes have not been enroled yet
21+
set_bifrost_maintenance: True
22+
23+
tasks:
24+
- name: Set maintenance mode
25+
command: |-
26+
docker exec bifrost_deploy bash -c '
27+
OS_CLOUD=bifrost openstack baremetal node maintenance set \
28+
--reason "Running drac-firmware-update.yml" \
29+
{{ inventory_hostname }}'
30+
become: true
31+
delegate_to: "{{ groups.seed.0 }}"
32+
vars:
33+
ansible_connection: ssh
34+
ansible_host: "{{ hostvars[groups.seed.0].ansible_host }}"
35+
when: set_bifrost_maintenance | bool
36+
37+
- name: Update firmware from repository on a HTTP
38+
idrac_firmware:
39+
idrac_ip: "{{ ipmi_address }}"
40+
idrac_user: "{{ secrets_ipmi_username }}"
41+
idrac_password: "{{ secrets_ipmi_password }}"
42+
reboot: True
43+
job_wait: True
44+
apply_update: "{{ dell_drm_apply_update | bool }}"
45+
share_name: "{{ dell_drm_address }}"
46+
catalog_file_name: "{{ dell_drm_repo }}"
47+
ignore_cert_warning: True
48+
register: idrac_firmware_output
49+
delegate_to: localhost
50+
when: ipmi_address is defined
51+
- debug:
52+
msg: "{{ idrac_firmware_output }}"
53+
54+
- name: Unset maintenance mode
55+
command: |-
56+
docker exec bifrost_deploy bash -c '
57+
OS_CLOUD=bifrost openstack baremetal node maintenance unset \
58+
{{ inventory_hostname }}'
59+
become: true
60+
delegate_to: "{{ groups.seed.0 }}"
61+
vars:
62+
ansible_connection: ssh
63+
ansible_host: "{{ hostvars[groups.seed.0].ansible_host }}"
64+
when: set_bifrost_maintenance | bool

‎etc/kayobe/ansible/requirements.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
---
22
collections:
3+
- name: dellemc.openmanage
4+
version: 4.2.0
35
- name: stackhpc.cephadm
46
version: 1.15.1
57
# NOTE: Pinning pulp.squeezer to 0.0.13 because 0.0.14+ depends on the

0 commit comments

Comments
 (0)