Improve database management documentation

Author: withinfocus

docs/contributing/database-migrations/_category_.yml

@@ -0,0 +1,2 @@
+label: "Database Migrations"
+position: 2

docs/contributing/database-migrations/edd.mdx

@@ -265,8 +265,7 @@ will not be run until the start of the next deploy when they are moved into `DbS
 After this deploy, to prep for the next release, all migrations in `DbScripts_transition` are moved
 to `DbScripts` and then all migrations in `DbScripts_finalization` are moved to `DbScripts`,
 conserving their execution order for a clean install. For the current branching strategy, PRs will
-be open against `main` when `rc` is cut to prep for this release. This PR automation will also
-handle renaming the migration file and updating any reference of `[dbo_finalization]` to `[dbo]`.
+be open against `main` when `rc` is cut to prep for this release.
 
 The next deploy will pick up the newly added migrations in `DbScripts` and set the previously
 repeatable _Transition_ migrations to no longer be repeatable, execute the _Finalization_

docs/contributing/database-migrations/ef.md

@@ -0,0 +1,30 @@
+---
+sidebar_position: 2
+---
+
+# Entity Framework
+
+:::info
+
+For instructions on how to apply database migrations, please refer to the
+[Getting Started](../../getting-started/server/database/ef/index.mdx) documentation.
+
+:::
+
+If you alter the database schema, you must create an EF migration script to ensure that EF databases
+keep pace with these changes. Developers must do this and include the migrations with their PR.
+
+To create these scripts, you must first update your data model in `Core/Entities` as desired. This
+will be used to generate the migrations for each of our EF targets. Additionally, for table changes
+it is strongly recommended to define or update an `IEntityTypeConfiguration<T>` to accurately
+represent any constraints needed on the data model.
+
+Once the model is updated, navigate to the `dev` directory in the `server` repo and execute the
+`ef_migrate.ps1` PowerShell command. You should provide a name for the migration as the only
+parameter:
+
+```bash
+pwsh ef_migrate.ps1 [NAME_OF_MIGRATION]
+```
+
+This will generate the migrations, which should then be included in your PR.

docs/contributing/database-migrations/index.md

@@ -0,0 +1,82 @@
+---
+sidebar_position: 1
+---
+
+# MSSQL
+
+:::info
+
+For instructions on how to apply database migrations, please refer to the
+[Getting Started](../../getting-started/server/database/mssql/index.md) documentation.
+
+:::
+
+:::tip
+
+We recommend reading [T-SQL Code Style](../code-style/sql.md) since it has a major impact in how we
+write migrations.
+
+:::
+
+## SQL database project
+
+The separate database definitions in `src/Sql/.../dbo` serve as a "master" reference for the
+intended and final state of the database at that time. This is crucial because the state of database
+definitions at the current moment may differ from when a migration was added in the past. These
+definitions act as a lint and validation step to ensure that migrations work as expected, and the
+separation helps maintain clarity and accuracy in database schema management and synchronization
+processes.
+
+Additionally, a
+[SQL database project](https://learn.microsoft.com/en-us/azure-data-studio/extensions/sql-database-project-extension-sdk-style-projects)
+is in place; however, instead of using the auto-generated migrations from
+[DAC](https://learn.microsoft.com/en-us/sql/relational-databases/data-tier-applications/data-tier-applications?view=sql-server-ver16),
+we manually write migrations. This approach is chosen to enhance performance and prevent accidental
+data loss, which is why we have both a `sqlproj` and standalone migrations.
+
+## Modifying the database
+
+In accordance with the tenets of [Evolutionary Database Design](./edd.mdx) every change must be
+considered as split into two parts:
+
+1. A backwards-compatible transition migration
+2. A non-backwards-compatible final migration
+
+It is likely that a change does not require a non-backwards-compatible end phase e.g. all changes
+may be backwards-compatible in their final form; in that case, only one phase of changes is
+required. With the use of beta testing, partial roll-outs, [feature flags](../feature-flags.md),
+etc. the often-chosen path is to spread a change across several major releases with a calculated
+future state that can perform a "cleanup" migration that is backwards-compatible but still
+represents an overall-_incompatible_ change beyond the boundaries of what we need for individual
+release safety.
+
+### Backwards compatible migration
+
+1. Modify the source `.sql` files in `src/Sql/dbo`.
+2. Write a migration script, and place it in `util/Migrator/DbScripts`. Each script must be prefixed
+   with the current date.
+
+Please take care to ensure:
+
+- any existing stored procedure accepts the same input parameters and that new parameters have
+  nullable defaults
+- when a column is renamed the existing stored procedures first check (coalesce) the new location
+  before falling back to the old location
+- continued updating of the old data columns since in case of a rollback no data should be lost
+
+### Non-backwards compatible migration
+
+These changes should be written from the perspective of "all data has been migrated" and any old
+stored procedures that were kept around for backwards compatibility should be removed. Any logic for
+syncing old and new data should also be removed in this step.
+
+1. Remove the backwards compatibility that is no longer needed.
+2. Write a new Migration and place it in `src/Migrator/DbScripts_finalization`. Name it
+   `YYYY-0M-FinalizationMigration.sql`.
+   - Typically migrations are designed to be run in sequence. However since the migrations in
+     `DbScripts_finalization` can be run out of order, care must be taken to ensure they remain
+     compatible with the changes to `DbScripts`. In order to achieve this we only keep a single
+     migration, which executes all backwards incompatible schema changes.
+
+Upon execution any finalization scripts will be [automatically moved](./edd.mdx#Online-environments)
+for proper history.

docs/contributing/database-migrations/mssql.md

@@ -40,9 +40,9 @@ each.
 
 Our EF implementations currently support Postgres, MySQL, and SQLite3.
 
-## Setting Up EF Databases
+## Creating the database
 
-The workflow here is broadly the same as with the normal MSSQL implementation: set up the docker
+The workflow here is broadly the same as with the normal MSSQL implementation: set up the Docker
 container, configure user secrets, and run migrations against their relating databases in
 chronological order.
 
@@ -139,7 +139,7 @@ that the changes take effect.
 
 :::
 
-### Migrations
+## Updating the database
 
 <Tabs
     groupId="provider"
@@ -185,7 +185,7 @@ You can also run migrations for all database providers at once using
 pwsh migrate.ps1 -all
 ```
 
-### Optional: Verify
+### Verifying changes
 
 If you would like to verify that everything worked correctly:
 
@@ -194,7 +194,7 @@ If you would like to verify that everything worked correctly:
   - Note: this requires a configured MSSQL database. You may also need to set up other EF providers
     for tests to pass.
 
-## Testing EF Changes
+## Testing changes
 
 In your `server/dev/secrets.json` file find or add this block of secrets in the root of the json
 structure:
@@ -230,3 +230,8 @@ existing databases if you have not already. If these settings are not present at
 With connection strings applied to your projects: ensure your databases are all migrated using
 `pwsh server/dev/migrate.ps1 --all`. Then you can run EF tests from the
 `test/Infrastructure.IntegrationTest` folder using `dotnet test`.
+
+## Modifying the database
+
+The process for modifying the database is described in
+[Migrations](./../../../../contributing/database-migrations/ef.md).

docs/getting-started/server/database/ef/index.mdx

@@ -23,4 +23,4 @@ of your database.
 ## Modifying the database
 
 The process for modifying the database is described in
-[Migrations](./../../../../contributing/database-migrations/).
+[Migrations](./../../../../contributing/database-migrations/mssql.md).