Recommended upgrade and migration path from Petrosian (16) to Radjabov (18)

[Jonathan-Caruana]

Hello ManageIQ community,

I’m looking for guidance on the recommended upgrade and migration strategy for ManageIQ.

Current setup

• Production
• ManageIQ: Petrosian (16)
• OS: CentOS Stream 8
• DB: PostgreSQL 13.7 (vmdb_production)

Target

• ManageIQ: Radjabov (18)
• OS: CentOS Stream 9
• DB: PostgreSQL 13.23 (vmdb_production)
• Fresh installation

Options considered so far

At this stage, I currently see two possible options:

1. In-place ManageIQ upgrade on CentOS Stream 8

• Upgrade Petrosian → Quinteros → Radjabov on CentOS Stream 8
• Even though the changelog states “CentOS Stream 8 is no longer supported”, it’s unclear whether this fully blocks installation or only means it is no longer tested/supported
• Once upgraded to Radjabov, perform a database dump/restore onto a fresh CentOS Stream 9 + Radjabov installation

2. OS-first approach

• Install CentOS Stream 9 with Petrosian
• Migrate the database from the current production system
• Then perform the ManageIQ upgrade path up to Radjabov on CentOS Stream 9

I would appreciate feedback on whether either of these approaches is recommended, supported, or strongly discouraged.

Regards,

[jrafanie]

• Even though the changelog states “CentOS Stream 8 is no longer supported”, it’s unclear whether this fully blocks installation or only means it is no longer tested/supported

In terms of tested/supported, yes, that's basically it. If you run into issues it will be in c extension gems that depend on OS packages. If we upgraded a c extension gem that depends on OS packages, like qpid-proton or pg, you risk not being able to install them. Also, things like the black console configuration page may run functions that may not be compatible with centos8 versions of packages. So as soon as you try to run radjabov code on centos 8, you could have issues if the correct dependencies aren't installed.

You can see some of the changes we did to get to el9 but keep in mind there could have been other changes we did afterwards: ManageIQ/manageiq-appliance-build#515

Once upgraded to Radjabov, perform a database dump/restore onto a fresh CentOS Stream 9 + Radjabov installation

Keep in mind, you'll need to transfer encryption keys (v2_key, or you won't be able to decrypt your encrypted passwords) and GUID (if you want the new server to take the place of the original one including all of it's settings).

Personally, I prefer the OS-first approach. In a test environment, install a new centos 9 petrosian, perform the db dump/restore, transfer the v2_key and GUID from the original server, migrate the database, reset the ae datastore to petrosian, configure it properly, and verify the installation including any automations.

Keep in mind, check the ruby and rails versions before and after. I suspect you're going from ruby 3.0 and rails 6.1 to ruby 3.1 and rails 7.0. If you run into any problems you can certainly review this list and see if you have your own customizations that need similar fixes:

ruby 3.1: #22696
rails 7.0: #22052

[Jonathan-Caruana]

Hello @jrafanie !

Many thanks for you quick and complete response.

Do you have a quick procedure to do this :

transfer the v2_key and GUID from the original server, migrate the database, reset the ae datastore to petrosian

I will try the OS-first approach like you suggested.

Regards,

[Jonathan-Caruana]

I suppose you were talking about thiss :

/var/www/miq/vmdb/GUID
/var/www/miq/vmdb/certs/v2_key

and then :

bundle exec rake evm:automate:reset

[Fryguy]

@Jonathan-Caruana Yes that's correct.

[jrafanie]

Thanks @Fryguy! @Jonathan-Caruana do this in a test environment and keep the existing setup for verification as you do the various steps. It's entirely possible something could be broken in the existing setup that you didn't noticed but because you're looking at things more in depth, you might find it now. It's much easier to fix things if we know if it was broken before or after the upgrade.

[jrafanie]

Some people will even do this over a span of a few days so that they can take a more recent db dump when they get the process written down and working and do it for "real" a bit later. They will setup a side by side environment where both systems are up and running. Keep in mind, both environments will connect to the various providers so you may want to make sure that won't be a problem. You can always change credentials on the "new" environment's providers to ensure they don't hit the providers until you're ready to test that.

As always, it's based on the environment so make sure you verify this side by side setup is fine for you as you may need to use a new server's hostname temporarily or permanently and make sure that matches any SSL certificates you may have setup previously.

[Jonathan-Caruana]

Hi @jrafanie @Fryguy

I am currently blocked :

1. Petrosian installation on COS 9

• Attempting a fresh installation of Petrosian on COS 9 is currently impossible, because no Petrosian packages are available for EL9/COS 9.
• Without the packages, I cannot proceed with installation or service startup, so no further testing on COS 9 has been possible.

2. Upgrade from Petrosian (on my actual COS 8 instance ) to Quinteros

• Attempting the upgrade fails with the same ActiveSupport/Logger and Bootsnap errors.
• The EVM server cannot start (evmserverd.service) immediately after systemd launch.
• Logs show:

uninitialized constant ActiveSupport::LoggerThreadSafeLevel::Logger (NameError)
from active_support/logger_thread_safe_level.rb

• Clearing tmp/cache and tmp/cache/bootsnap* does not resolve the issue.
• DISABLE_BOOTSNAP=1 allows the server to attempt startup, but then multiple standard library gems are reported missing:

Could not find cgi-0.5.1, forwardable-1.4.0, stringio-3.2.0, net-protocol-0.2.2, timeout-0.6.0

• bundle check confirms all Gemfile dependencies are installed, suggesting a bundler/environment mismatch.
• System ruby is 3.0.4p208; ruby -v and bundle exec ruby -v are consistent.
• Removing/reinstalling the bundle does not solve the issue.
• Attempts to start EVM manually (sudo -u manageiq bundle exec ruby lib/workers/bin/evm_server.rb) fail with the same errors.

3. Systemd service behavior

• systemctl status evmserverd.service shows repeated exit code 1 failures and rapid restart attempts:

Main process exited, code=exited, status=1/FAILURE
Service RestartSec=100ms expired, scheduling restart

• Journal logs confirm the NameError in ActiveSupport every attempt.

4. Steps already attempted

• Removed all Bootsnap cache (tmp/cache/bootsnap* and tmp/cache).
• Reinstalled all gems using bundle install and bundle pristine.
• Verified Ruby version consistency.
• Attempted disabling Bootsnap (DISABLE_BOOTSNAP=1) to bypass load cache.
• Verified standard library availability (ruby -r logger -e 'puts Logger' works).

At this point, both installing Petrosian on COS 9 and upgrading Petrosian to Radjabov on COS 8 are blocked due to gem/load path issues with ActiveSupport and Bootsnap.

Any guidance on resolving these dependency/load path issues or alternate upgrade/install procedures would be greatly appreciated.

Thanks,

Jonathan

[jrafanie]

@Jonathan-Caruana Sorry, I wasn't precise. What I'm advising is you deploy a new Radjabov appliance from https://www.manageiq.org/download/ and transfer the GUID, v2_key and import the database, migrate it, reset the automation domain, etc.

While it's possible to upgrade an old appliance to a new one, you'd have to deal with each of the little things we had to fix to get to radjabov. If you take a radjabov appliance, you can bring over the identity (GUID) and encryption/decryption key (v2_key), and you should inherit the same settings as previously done. Note, your hostname will be different so you may have to verify the appliance's information is correct when you do the production upgrade as you may have hostnames in ssl certificates that may not match up. The solution would be to either do new certificates or bring down the old system as you bring up the new one with the old one's name. I've seen users who had automation that connected to an appliance by hostname or ipaddress as part of some automation, so keep that in mind if the new system doesn't match the old hostname/ipaddress.

[Jonathan-Caruana]

Hello @jrafanie

My worry is about to transfer the database from Petrosian directly to Radjabov. I my mind it's couldn't work because the database probably need to be upgrade no?

It's the only reason why i try to follow the upgrade path Petrosian -> Quinteros -> Radjabov.

For now, i can't install Petrosian on COS 9 (i didn't find any package on your repo to do this) to perform on db dump/restore on same version and i'm unable to upgrade from Petrosian to Radjabov on my COS 8 appliance.

i feel like i'm stuck :p

[jrafanie]

So, petrosian first shipped with PG 10 but I think we moved to postgresql 13 with the more recent tags. Radjabov is still postgresql 13 so that's the least of my concerns. If you're using HA postgresql or region replication, it will require a little more configuration, such as setting up the HA nodes again, or reconfiguring replication but it's still not jumping versions.

[Fryguy]

My worry is about to transfer the database from Petrosian directly to Radjabov. I my mind it's couldn't work because the database probably need to be upgrade no?

From the opening post, you're going from PG 13 to PG 13, so there's no version difference there. Backup on the petrosian appliance, copy to the radjabov appliance, and then restore on the radajabov appliance. Then you can run bundle exec rake db:migrate to get the database migrated to the latest version. (I thought these steps were documented - @bdunne?)

[jrafanie]

(I thought these steps were documented - @bdunne?)

We have a few discussions where we mentioned some of these procedures: https://github.com/orgs/ManageIQ/discussions/22295#discussioncomment-4657845 (note, there is no dnf changes in this discussion to worry about since that discussion was going from one ruby and postgresql version to another) and https://github.com/orgs/ManageIQ/discussions/22745#discussioncomment-8170279. Note, if you do HA/region replication (multiple postgresql databases), you'll need to reconfigure them afterwards as we're not exporting the cluster, only the database.

[Jonathan-Caruana]

Hello @jrafanie @Fryguy,

I followed your previous instructions as closely as possible. Below is a summary of what I’ve done and the issue I’m currently facing.

What I did

• Installed Radjabov on a new CentOS Stream 9 system

• Copied the full dump of vmdb_production from my CentOS Stream 8 / Petrosian instance and imported it

• Copied the following files from the old appliance:

  • /var/www/miq/vmdb/GUID
  • /var/www/miq/vmdb/certs/v2_key
  • /var/www/miq/vmdb/config/database.yml

• Generated a new SSL certificate including the new FQDN

• Started evmserverd

• Ran:

  • bundle exec rake db:migrate
  • bundle exec rake evm:automate:reset

Current status

• evmserverd, postgresql, and httpd are all running
• Accessing the UI returns a 503 error

When checking the Apache logs, I see repeated proxy errors indicating connection refusals on ports 3000–3009, for example:

Connection refused: AH00957: http: attempt to connect to 0.0.0.0:3000 failed
AH00959: ap_proxy_connect_backend disabling worker for (0.0.0.0:3000)

This makes it look like Puma is not running properly, or at least not listening on the expected 300* ports.

Additional tests

I tried starting the Rails server manually with:

bundle exec rails server -b 0.0.0.0 -p 3000 -e production

In this case, the UI is reachable and works as expected.

Question

Did I miss something related to Puma configuration or initialization? As far as I remember, I never had to manually touch Puma on my previous appliance.

One additional detail that may or may not be relevant: to align with our internal standards, I moved /var/www/miq to /opt/apps/miq and set up a bind mount so that /var/www/miq still exists and points to the new location. Could this be related to the issue?

Thanks in advance for your help.

[jrafanie]

• Installed Radjabov on a new CentOS Stream 9 system

is this a radjabov appliance from https://www.manageiq.org/download/? If not, there could be some setup that we do in the appliance that was not done. It's a lot: https://github.com/ManageIQ/manageiq-appliance and https://github.com/ManageIQ/manageiq-appliance-build/blob/master/kickstarts/base.ks.erb

Either way, you'll need to look at the logs. It's possible there was an error in the UI workers at startup or something else preventing it from working.

[Jonathan-Caruana]

I’m not entirely sure whether I’m running the very latest build, but this is how I installed the appliance:

dnf install https://rpm.manageiq.org/release/18-radjabov/el9/noarch/manageiq-release-18.0-1.el9.noarch.rpm
dnf module enable ruby:3.3 -y
dnf makecache
dnf install -y manageiq-appliance

Here is the list of installed ManageIQ packages:

manageiq-release-18.0-1.el9.noarch
manageiq-ansible-venv-18.1.0-20250408024349.el9.x86_64
manageiq-system-18.1.0-20250408024349.el9.x86_64
manageiq-gemset-18.1.0-20250408024349.el9.x86_64
manageiq-core-18.1.0-20250408024349.el9.x86_64
manageiq-core-services-18.1.0-20250408024349.el9.x86_64
manageiq-ui-18.1.0-20250408024349.el9.x86_64
manageiq-gemset-services-18.1.0-20250408024349.el9.x86_64
manageiq-appliance-18.1.0-20250408024349.el9.x86_64

However, thanks to the logs, I eventually identified why Puma was failing to start:

Dalli::RingError: No server available

After some investigation, this turned out to be related to the memcached service. By default, memcached tries to listen on both 127.0.0.1 and ::1. In our environment, IPv6 is disabled by default when deploying new virtual servers, which caused the issue.

In any case, thank you for your support — it was very helpful. I’m now going to take a few days to perform a thorough review of our entire setup.

Regards,

[Fryguy]

Oh that's very interesting about memcached on both ports. Can you open a bug ticket for that in https://github.com/ManageIQ/manageiq/issues?

[Jonathan-Caruana]

@Fryguy

It's done 👍 : #23695