[3.0] Adds migration and cleanup steps for upgrading from old versions#9120
[3.0] Adds migration and cleanup steps for upgrading from old versions#9120Sesquipedalian wants to merge 7 commits intoSimpleMachines:release-3.0from
Conversation
Sesquipedalian
changed the title
Sesquipedalian
force-pushed
the
3.0/upgrade_old_versions
branch
from
a2e1ec0 to
ff9e7b5
Compare
|
First attempt at running an upgrade with this PR tonight. First test was a 2.1 to 3.0 upgrade. I had to do a composer update to get this to work. I actually wonder if the /vendor contents & composer lock should have been in the commits??? Not sure what the proper protocol should be. I was able to get past that by committing the update to composer.lock locally in my test branch for this PR. Next problem is that the 'Continue' buttons are all associated with 3 second countdowns. I don't think there should be any countdown at all... That wasn't even enough time to type in my credentials. And it shouldn't continue until I say so... I got past that by updating the upgrade template countdown to 30. And reading and typing quickly. Next problem is that it appears to have automatically started trying to do a postgresql upgrade on my mysql database. Screenshot below. I wasn't able to get past this. Not sure why it's attempting pg, I didn't even have pg started. And the 2.1 style Settings.php file specifies $db_type = 'mysql'; Note that with this screen, it's still doing an auto-continue every 3 seconds: Mysql 8.4.4 |
|
Well, that's a bunch of weird surprises. I didn't touch any of those things in this PR. But I guess we'll need to fix them so you can test. |
I'm not sure why you had to do that. There are no differences between this branch and the release-3.0 branch for composer.lock, nor any for composer.json.
I could have sworn that I removed that in an earlier PR.
That's extremely weird. More investigation is needed before I can even begin to guess why you saw that. |
I believe the latest commit has fixed that one. It appears to have been introduced in #9090. |
Oh, now I see. The screenshot doesn't indicate that the upgrader thought your database was PostgreSQL. It is just showing that the upgrader was walking through some of the early migration substeps that apply only to PostgreSQL (and skipping them because they weren't applicable) when it encountered an error with the message "TypeError: NetworkError when attempting to fetch resource." The question is, why did it encounter that error? Based on a quick web search, it indicates a CORS problem while making an HTTP request, which is rather odd in this context. To help us figure where the heck that error is coming from, please check:
|
|
Good news: I refreshed the code from the latest this morning, & the 2.1 => 3.0 upgrade went well! I found that the composer.lock was simply because I'd run the composer update & there is a new version of fixer available. Side note: I haven't had a successful 3.0 installer run in a very long time (I'd logged several issues early 2025). I still cannot get it to run, with or without this PR. Logged #9131 I just attempted a 2.0 (latin1) => 3.0 upgrade & got the following error. All files are there, I believe, & I double-checked the files used in the file check, and the versions look good. Haven't dug much deeper; I see the code looks for some classes also, it might be failing there. A 2.0 (utf8) => 3.0 upgrade attempt has the same result. 
|
|
I attempted a 1.1 => 3.0 upgrade & got a WSOD with the following:
Pretty sure custom avatars didn't exist before 2.0? |
|
I attempted a 1.0 => 3.0 upgrade & got a different WSOD:
|
|
yabbse:
It's been a while since I've run these, always possible I'm grappling with environment issues. |
|
I took a peek at the 2.0 issue. First time seeing this code, so take this with a grain of salt... It finds all the files OK, Upgrade.php line 704. When it starts looking for the 2.0 classes is where there is an issue, Upgrade.php, line 718. It finds & loads the first one fine (SMF\Maintenance\Migration\v2_0\PostgreSQLFunctions), and loadClass() is properly returning true. So far, so good... But for some reason it still throws the exception on line 719. I believe it's falling back thru a closure & losing the "true" there. So, well hidden under the covers, it is throwing the error: Exception->__construct($message = 'SMF\Maintenance\Migration\v2_0\PostgreSQLFunctions does not exist') D:\wamp64\www\van2021\Sources\Maintenance\Tools\Upgrade.php:719 When in fact it was loaded OK. |
|
Note that loadClass actually does an include, which in turn, recursively includes MigrationBase & SubStepInterface immediately after PostgreSQLFunctions. It's possible there's an issue with one of those? Topic me, all appear to load ok. |
Sesquipedalian
force-pushed
the
3.0/upgrade_old_versions
branch
from
356c8c9 to
ddf782e
Compare
Just trying to clarify here. When you write "loadClass", do you mean "class_exists"? |
|
class_exists by default forces an autoload, which calls Composer's loadClass as well as many layers of other stuff... A fractal recursive maze of OOPsie. check_exists loads everything. If that's not desired, I think you can pass false as 2nd param. I'm not sure why that's failing though. Seems to be loading fine - including the layers of base classes & interfaces. Makes no sense. |
|
I think I found my new forum signature: "A fractal recursive maze of OOPsie." |
|
The last class loaded was SubStepInterface. This might make sense if there were an issue with SubStepInterface, but I don't see one. |
|
Gotcha. Hm, that's still odd. I don't have any ideas about it yet. I'll have to try again to see if I can reproduce the error locally. |
|
OK, some progress... My version of Composer was a few months old, 2.8.8. Latest version is 2.9.5, and the error was clearly deep in the heart of Composer, so... I did a composer self-update to 2.9.5, and now I'm getting further... But the error now is identical to the 1.1 error. And yes, I refreshed to get the most recent commits:
|
|
Odd the 2.1 => 3.0 upgrade was not impacted, but the 2.0 => 3.0 upgrade was... |
That should be an easy fix. I'll push a commit for that later tonight. |
|
Does that latest commit allow installation to proceed, @sbulen? |
|
Sorry, wasn't sure you wanted a retest yet. Reran 2.1 => 3.0 - OK Latin1:
utf8:
|
Sesquipedalian
force-pushed
the
3.0/upgrade_old_versions
branch
2 times, most recently
from
69a55d1 to
d92af01
Compare
|
I've now pushed up fixes for all issues reported so far. When you are able, @sbulen, please check out a new copy of this brach and try again. |
Should be fixed in latest commits.
Can you try again with debug enabled so that we can see where the problem lies?
Should be fixed in latest commits.
Should be fixed in latest commits. |
|
Note I see this in the PHP log... Should have looked at this earlier... This includes errors from all yesterday's tests...
|
|
OK today's tests! Another timer nit... Timer looks better, but only increments after major steps are completed. This in the past was an indicator it was actually running, even with debug off. In fact, it said so on the screen... "The time elapsed increments from the server to show progress is being made!" That was helpful on large forum upgrades. I'm asking for debug on on all these, since that seems to provide better info. 2.1:
*** Nothing in the php/mysql/apache logs... This is the bottom of the upgrader log:
2.0 latin1:
PHP log: 2.0 utf8:
PHP log: 1.1:
PHP log: |
Sesquipedalian
force-pushed
the
3.0/upgrade_old_versions
branch
from
b1f4238 to
d9dceb2
Compare
These three should be fixed in the latest commits. |
Please temporarily replace the contents of ./Sources/Maintenance/Migrations/v3_0/RecurringEvents.php with the following. Like before, it will dump a mess into your install.log that should help me figure out the problem. <?php
/**
* Simple Machines Forum (SMF)
*
* @package SMF
* @author Simple Machines https://www.simplemachines.org
* @copyright 2026 Simple Machines and individual contributors
* @license https://www.simplemachines.org/about/smf/license.php BSD
*
* @version 3.0 Alpha 4
*/
declare(strict_types=1);
namespace SMF\Maintenance\Migration\v3_0;
use SMF\Db\DatabaseApi as Db;
use SMF\Db\Schema;
use SMF\Maintenance\Migration\MigrationBase;
class RecurringEvents extends MigrationBase
{
/*******************
* Public properties
*******************/
/**
*
*/
public string $name = 'Adding support for recurring events';
/****************
* Public methods
****************/
/**
*
*/
public function execute(): bool
{
Maintenance::$tool->logProgress('Getting calendar table definition', true);
$table = new Schema\v3_0\Calendar();
Maintenance::$tool->logProgress('Getting current calendar table structure', true);
$existing_structure = $table->getCurrentStructure();
Maintenance::$tool->logProgress('Adding missing columns', true);
foreach ($table->columns as $column) {
if (!isset($existing_structure['columns'][$column->name])) {
Maintenance::$tool->logProgress($column->name, true);
$table->addColumn($column);
}
$this->handleTimeout();
}
if (Db::$db->title === MYSQL_TITLE) {
Maintenance::$tool->logProgress('Rearranging columns', true);
Db::$db->query(
'ALTER TABLE {db_prefix}calendar
MODIFY COLUMN start_date DATE AFTER id_member',
[],
);
Db::$db->query(
'ALTER TABLE {db_prefix}calendar
MODIFY COLUMN end_date DATE AFTER start_date',
[],
);
}
Maintenance::$tool->logProgress('Building updates', true);
$updates = [];
$request = Db::$db->query(
'SELECT id_event, start_date, end_date, start_time, end_time, timezone
FROM {db_prefix}calendar',
[],
);
while ($row = Db::$db->fetch_assoc($request)) {
try {
$row = array_diff($row, array_filter($row, 'is_null'));
$allday = (
!isset($row['start_time'])
|| !isset($row['end_time'])
|| !isset($row['timezone'])
|| !\in_array($row['timezone'], timezone_identifiers_list(\DateTimeZone::ALL_WITH_BC))
);
$start = new \DateTime($row['start_date'] . (!$allday ? ' ' . $row['start_time'] . ' ' . $row['timezone'] : ''));
$end = new \DateTime($row['end_date'] . (!$allday ? ' ' . $row['end_time'] . ' ' . $row['timezone'] : ''));
if ($allday) {
$end->modify('+1 day');
}
$duration = date_diff($start, $end);
$format = '';
foreach (['y', 'm', 'd', 'h', 'i', 's'] as $part) {
if ($part === 'h') {
$format .= 'T';
}
if (!empty($duration->{$part})) {
$format .= '%' . $part . ($part === 'i' ? 'M' : strtoupper($part));
}
}
$format = rtrim('P' . $format, 'PT');
// TODO: Fix it right.
if ((int) $end->format('Y') > 9999) {
$end->setDate(9999, (int) $end->format('m'), (int) $end->format('d'));
}
$updates[$row['id_event']] = [
'id_event' => $row['id_event'],
'duration' => $duration->format($format),
'end_date' => $end->format('Y-m-d'),
'rrule' => 'FREQ=YEARLY;COUNT=1',
];
} catch (\Throwable $e) {
Maintenance::$tool->logProgress($e->getMessage(), true);
throw $e;
}
}
Db::$db->free_result($request);
foreach ($updates as $id_event => $changes) {
Maintenance::$tool->logProgress('Submitting update: ' . \SMF\Utils::normalizeSpaces(\SMF\Config::varExport($changes), true, true, ['nobreaks' => true]), true);
Db::$db->query(
'UPDATE {db_prefix}calendar
SET duration = {string:duration}, end_date = {date:end_date}, rrule = {string:rrule}
WHERE id_event = {int:id_event}',
$changes,
);
}
Maintenance::$tool->logProgress('Removing end_time column', true);
Db::$db->remove_column('{db_prefix}calendar', 'end_time');
Maintenance::$tool->logProgress('returning', true);
return true;
}
} |
Sesquipedalian
force-pushed
the
3.0/upgrade_old_versions
branch
4 times, most recently
from
a1f0fab to
ac9f699
Compare
|
Today's results. The 2.1 execution included the changes requested above. I think the error reporting in the 2.1 execution should be permanent & universal. Errors are visible in the browser, and also logged in the upgrader log. That's what's needed. The output from the other executions is useless. 2.1:
system logs: Upgrader log, bottom:
2.0 latin1:
system logs: Upgrader log, bottom:
2.1 utf8:
system logs: Upgrader log, complete:
1.1:
system logs: Upgrader log, complete:
|
Derp. That's because of a silly mistake on my part. Please try again with this version: <?php
/**
* Simple Machines Forum (SMF)
*
* @package SMF
* @author Simple Machines https://www.simplemachines.org
* @copyright 2026 Simple Machines and individual contributors
* @license https://www.simplemachines.org/about/smf/license.php BSD
*
* @version 3.0 Alpha 4
*/
declare(strict_types=1);
namespace SMF\Maintenance\Migration\v3_0;
use SMF\Db\DatabaseApi as Db;
use SMF\Db\Schema;
use SMF\Maintenance\Migration\MigrationBase;
class RecurringEvents extends MigrationBase
{
/*******************
* Public properties
*******************/
/**
*
*/
public string $name = 'Adding support for recurring events';
/****************
* Public methods
****************/
/**
*
*/
public function execute(): bool
{
\SMF\Maintenance\Maintenance::$tool->logProgress('Getting calendar table definition', true);
$table = new Schema\v3_0\Calendar();
\SMF\Maintenance\Maintenance::$tool->logProgress('Getting current calendar table structure', true);
$existing_structure = $table->getCurrentStructure();
\SMF\Maintenance\Maintenance::$tool->logProgress('Adding missing columns', true);
foreach ($table->columns as $column) {
if (!isset($existing_structure['columns'][$column->name])) {
\SMF\Maintenance\Maintenance::$tool->logProgress($column->name, true);
$table->addColumn($column);
}
$this->handleTimeout();
}
if (Db::$db->title === MYSQL_TITLE) {
\SMF\Maintenance\Maintenance::$tool->logProgress('Rearranging columns', true);
Db::$db->query(
'ALTER TABLE {db_prefix}calendar
MODIFY COLUMN start_date DATE AFTER id_member',
[],
);
Db::$db->query(
'ALTER TABLE {db_prefix}calendar
MODIFY COLUMN end_date DATE AFTER start_date',
[],
);
}
\SMF\Maintenance\Maintenance::$tool->logProgress('Building updates', true);
$updates = [];
$request = Db::$db->query(
'SELECT id_event, start_date, end_date, start_time, end_time, timezone
FROM {db_prefix}calendar',
[],
);
while ($row = Db::$db->fetch_assoc($request)) {
try {
$row = array_diff($row, array_filter($row, 'is_null'));
$allday = (
!isset($row['start_time'])
|| !isset($row['end_time'])
|| !isset($row['timezone'])
|| !\in_array($row['timezone'], timezone_identifiers_list(\DateTimeZone::ALL_WITH_BC))
);
$start = new \DateTime($row['start_date'] . (!$allday ? ' ' . $row['start_time'] . ' ' . $row['timezone'] : ''));
$end = new \DateTime($row['end_date'] . (!$allday ? ' ' . $row['end_time'] . ' ' . $row['timezone'] : ''));
if ($allday) {
$end->modify('+1 day');
}
$duration = date_diff($start, $end);
$format = '';
foreach (['y', 'm', 'd', 'h', 'i', 's'] as $part) {
if ($part === 'h') {
$format .= 'T';
}
if (!empty($duration->{$part})) {
$format .= '%' . $part . ($part === 'i' ? 'M' : strtoupper($part));
}
}
$format = rtrim('P' . $format, 'PT');
// TODO: Fix it right.
if ((int) $end->format('Y') > 9999) {
$end->setDate(9999, (int) $end->format('m'), (int) $end->format('d'));
}
$updates[$row['id_event']] = [
'id_event' => $row['id_event'],
'duration' => $duration->format($format),
'end_date' => $end->format('Y-m-d'),
'rrule' => 'FREQ=YEARLY;COUNT=1',
];
} catch (\Throwable $e) {
\SMF\Maintenance\Maintenance::$tool->logProgress($e->getMessage(), true);
throw $e;
}
}
Db::$db->free_result($request);
foreach ($updates as $id_event => $changes) {
\SMF\Maintenance\Maintenance::$tool->logProgress('Submitting update: ' . \SMF\Utils::normalizeSpaces(\SMF\Config::varExport($changes), true, true, ['nobreaks' => true]), true);
Db::$db->query(
'UPDATE {db_prefix}calendar
SET duration = {string:duration}, end_date = {date:end_date}, rrule = {string:rrule}
WHERE id_event = {int:id_event}',
$changes,
);
}
\SMF\Maintenance\Maintenance::$tool->logProgress('Removing end_time column', true);
Db::$db->remove_column('{db_prefix}calendar', 'end_time');
\SMF\Maintenance\Maintenance::$tool->logProgress('returning', true);
return true;
}
} |
|
Also, let's focus for now on getting the 2.1 → 3.0 upgrade all tested and working. Once that has all been fixed, we'll finally be able to move on to testing the original purpose of this PR, which was to support upgrading from older versions. |
Ok. Note there are layers of dependencies. It's kind of hard to test anything atm with #9146 & #9147 open, IMO. After so many huge changes, it feels like we need a tweakfest, to get some ironing done... I dont think it's a lot of work, but its important to get the basics working again. Without doing so, we dont know if issues encountered in testing were introduced by the upgrade or not. Then we can spend some time on 2.1=> 3.0 without this PR. Not too much, because those 2.1 => 3.0 tests need to be repeated with this PR. Then return to this PR. |
Sesquipedalian
force-pushed
the
3.0/upgrade_old_versions
branch
from
ac9f699 to
9e67b84
Compare
Hm. Yeah, I'm inclined to agree. I will move all the various bug fixes that have been added to this PR into their own, separate PR that we can test on their own. |
Sesquipedalian
force-pushed
the
3.0/upgrade_old_versions
branch
from
9e67b84 to
abb9135
Compare
|
Since #9162 needs to be resolved first, I won't review anything. But getting progress on 2,0, 1.x and yabbse is great. |
Sesquipedalian
force-pushed
the
3.0/upgrade_old_versions
branch
from
abb9135 to
54ff979
Compare
Signed-off-by: Jon Stovell <jonstovell@gmail.com>
Signed-off-by: Jon Stovell <jonstovell@gmail.com>
Signed-off-by: Jon Stovell <jonstovell@gmail.com>
Signed-off-by: Jon Stovell <jonstovell@gmail.com>
Signed-off-by: Jon Stovell <jonstovell@gmail.com>
Signed-off-by: Jon Stovell <jonstovell@gmail.com>
Signed-off-by: Jon Stovell <jonstovell@gmail.com>
Sesquipedalian
force-pushed
the
3.0/upgrade_old_versions
branch
from
54ff979 to
b17bdca
Compare
3 participants
I've had this sitting around nearly completed for months now, so I thought I'd take a few hours to finish it off and submit it.
This PR adds all the necessary table schemas, migration steps, and cleanup steps required to allow the upgrader to upgrade any version of SMF or even YaBB SE.
@jdarwood007, I know that we had previously discussed writing converters for old SMF versions rather than upgrader steps. However, when I sat down to do it, writing upgrader steps turned out to be much easier, because I could base them on the old upgrade scripts rather than trying to come up with entirely new logic. And since everything in the new SMF\Maintenance paradigm is so clearly defined and reliable, I don't think these new upgrader steps are going to be a burden to maintain the way the old upgrade scripts were.
@sbulen, is there any chance you could fire up that marvellous test suite of yours to make sure that these all work? Overall, I expect that it go well, but there are doubtless some copy-paste errors and typos hiding in there someplace.