diff --git a/charles-patching-guide.md b/charles-patching-guide.md index 306bb12..d1a9d10 100644 --- a/charles-patching-guide.md +++ b/charles-patching-guide.md @@ -1,10 +1,10 @@ # TopNotch Gaming — Patching Guide for Charles Tepper -**Version Number** 1.0.2 Release +**Version Number:** 1.0.3 Release -**Last updated:** 2026-05-23 +**Last updated:** 2026-05-26 -**Applies to:** All Salt-managed Linux hosts (15 minions) +**Applies to:** All Salt-managed Linux hosts (18 minions) **Run from:** undyne (192.168.16.9) — Salt Master @@ -12,24 +12,28 @@ ## Overview -Patching is handled by Salt and runs in four groups: **dev → staging → prod/Infra**. The orchestrator waits for each tier to succeed before moving to the next, so a bad update won't automatically roll forward to production. +Patching is handled by Salt and runs in five tiers: **dev → staging → prod → infra → master**. The orchestrator waits for each tier to succeed before moving to the next, so a bad update won't automatically roll forward. Reboots are **disabled by default** (controlled via pillar). The patch run upgrades packages only; reboots are a separate decision made after reviewing what changed. ---- -## Overview of Machines we manage using Salt - -INFRA: Granberia, Undyne, Midgar, Junon, gw5, Kalm, almaelma, pve, proxvm1, proxweb0, proxVM2 - -PROD: Leviathan, Gnome, Salamander, vaultwarden - -*DO NOTE: PVE, ProxVM1, ProxWeb0, and ProxVM2 are all the Proxmox Hypervisors. Under NO circumstance should these ever have any updates that are pertainent or require a reboot, unless strictly specified under the header "What's Held (Won't be Patched)"* - -INFRA should be rebooted every Sunday if a reboot is required. Patches should be applied every Sunday. - -PROD should be rebooted every FIRST sunday regardless of if a reboot is required or not. Patches should be applied every Sunday. +**Critical:** The `master` tier (undyne) patches last — only after all other tiers succeed. This ensures the Salt master is never taken down mid-rollout. If any tier fails, the rollout halts and undyne is never touched. --- + +## Overview of Machines We Manage Using Salt + +**INFRA:** Granberia, Undyne, Midgar, Junon, gw5, Kalm, almaelma, pve, proxvm1, proxweb0, proxVM2, cosmo, sylph + +**PROD:** Leviathan, Gnome, Salamander, vaultwarden, shiva + +*DO NOTE: PVE, ProxVM1, ProxWeb0, and ProxVM2 are all Proxmox Hypervisors. Under NO circumstance should these ever have any updates that are pertinent to or require a reboot, unless strictly specified under the header "What's Held (Won't Be Patched)".* + +INFRA should be rebooted every Sunday **if a reboot is required**. Patches should be applied every Sunday. + +PROD should be rebooted every **first Sunday** regardless of whether a reboot is required or not. Patches should be applied every Sunday. + +--- + ## Before You Start SSH into undyne using your AD credentials: @@ -103,19 +107,25 @@ Salt and Proxmox upgrades are handled separately and deliberately — not throug ## Hosts in Scope -The orchestrator runs all tiers automatically in sequence. The following hosts are Salt-managed and will be patched: +The orchestrator runs all tiers automatically in sequence: -`almaelma`, `granberia`, `undyne`, `leviathan`, `salamander`, `gnome`, `sylph`, `vaultwarden`, `cosmo`, `midgar`, `pve`, `proxVM1`, `proxVM2`, `ProxWeb0`, `junon`, `gw5 +| Tier | Hosts | +|------|-------| +| dev | `almaelma`, `sylph`, `gw5` | +| staging | `pve`, `proxVM1`, `proxVM2`, `ProxWeb0` | +| prod | `leviathan`, `gnome`, `salamander`, `vaultwarden`, `kalm`, `shiva` | +| infra | `cosmo`, `midgar`, `junon`, `granberia` | +| master | `undyne` | -**Not patched by Salt:** `npmp-topnotch` (Alpine, unmanaged), `shiva` (bootstrap failing), Windows hosts (luca, chocobo, WINSVR03). +**Not patched by Salt:** `npmp-topnotch` (Alpine, unmanaged), Windows hosts (luca, chocobo, WINSVR03). --- -## Special Notes about patch targets +## Special Notes About Patch Targets `leviathan`, `salamander`, and `gnome` are all AMP Instance Manager Production targets and are active production pieces. -The aim is not to reboot these unless ABSOLUTELY necessary, which after about a month, they -should- be rebooted anyway. If it's that time of the month (har har), notify `adupree`. +The aim is not to reboot these unless ABSOLUTELY necessary. After about a month of uptime, they *should* be rebooted regardless. If it's that time of the month (har har), notify `adupree`. ---