5.5 KiB
TopNotch Gaming — Patching Guide for Charles Tepper
Version Number: 1.0.3 Release
Last updated: 2026-05-26
Applies to: All Salt-managed Linux hosts (18 minions)
Run from: undyne (192.168.16.9) — Salt Master
Overview
Patching is handled by Salt and runs in five tiers: dev → staging → prod → infra → master. The orchestrator waits for each tier to succeed before moving to the next, so a bad update won't automatically roll forward.
Reboots are disabled by default (controlled via pillar). The patch run upgrades packages only; reboots are a separate decision made after reviewing what changed.
Critical: The master tier (undyne) patches last — only after all other tiers succeed. This ensures the Salt master is never taken down mid-rollout. If any tier fails, the rollout halts and undyne is never touched.
Overview of Machines We Manage Using Salt
INFRA: Granberia, Undyne, Midgar, Junon, gw5, Kalm, almaelma, pve, proxvm1, proxweb0, proxVM2, cosmo, sylph
PROD: Leviathan, Gnome, Salamander, vaultwarden, shiva
DO NOTE: PVE, ProxVM1, ProxWeb0, and ProxVM2 are all Proxmox Hypervisors. Under NO circumstance should these ever have any updates that are pertinent to or require a reboot, unless strictly specified under the header "What's Held (Won't Be Patched)".
INFRA should be rebooted every Sunday if a reboot is required. Patches should be applied every Sunday.
PROD should be rebooted every first Sunday regardless of whether a reboot is required or not. Patches should be applied every Sunday.
Before You Start
SSH into undyne using your AD credentials:
ssh charles@undyne.topnotch.gaming
Confirm Salt is healthy — all minions should return True:
sudo salt '*' test.ping
If any minion doesn't respond, investigate before patching. A minion that's down won't be patched and won't block the run, but you want to know about it.
Step 1 — Dry Run (Always Do This First)
sudo salt-run state.orch orch.patching test=True
This shows exactly what would be upgraded on every host without making any changes. Review the output for anything unexpected — kernel updates, Salt itself, or Proxmox packages should not appear (they're held). If they do, stop and flag it.
Step 2 — Run the Patch
Once you're happy with the dry run output:
sudo salt-run state.orch orch.patching
The orchestrator will work through tiers sequentially. Each tier must complete cleanly before the next begins. Total runtime varies — expect 5–15 minutes for a full fleet run.
Step 3 — Review Output
Look for any False results in the summary. A state returning False means a package update failed on that host. Note the hostname and package, then investigate directly on that host.
Reboot Policy
Reboots are not automatic. After patching, check if any hosts need a reboot (e.g. kernel update applied):
sudo salt '*' cmd.run 'test -f /var/run/reboot-required && echo REBOOT_NEEDED || echo ok'
If reboots are needed, coordinate timing and reboot hosts one at a time — especially for the Proxmox cluster nodes (pve, proxVM1, ProxWeb0). Never reboot more than one cluster node simultaneously or quorum is lost and the remaining nodes go read-only.
Standalone node proxVM2 can be rebooted independently without cluster impact.
What's Held (Won't Be Patched)
The following packages are pinned and will never be upgraded by the patching run:
- Proxmox stack:
pve-manager,proxmox-ve,proxmox-default-kernel,proxmox-kernel-6.14,proxmox-kernel-7.0 - Salt suite:
salt-api,salt-cloud,salt-common,salt-master,salt-minion,salt-ssh,salt-syndic
Salt and Proxmox upgrades are handled separately and deliberately — not through the automated patching run.
Hosts in Scope
The orchestrator runs all tiers automatically in sequence:
| Tier | Hosts |
|---|---|
| dev | almaelma, sylph, gw5 |
| staging | pve, proxVM1, proxVM2, ProxWeb0 |
| prod | leviathan, gnome, salamander, vaultwarden, kalm, shiva |
| infra | cosmo, midgar, junon, granberia |
| master | undyne |
Not patched by Salt: npmp-topnotch (Alpine, unmanaged), Windows hosts (luca, chocobo, WINSVR03).
Special Notes About Patch Targets
leviathan, salamander, and gnome are all AMP Instance Manager Production targets and are active production pieces.
The aim is not to reboot these unless ABSOLUTELY necessary. After about a month of uptime, they should be rebooted regardless. If it's that time of the month (har har), notify adupree.
If Something Goes Wrong
- Minion stops responding mid-run: SSH directly to the host and check
sudo systemctl status salt-minion. Restart it withsudo systemctl restart salt-minion, then re-run. - Salt itself was accidentally upgraded: Follow the Salt 3007.x rollback procedure — ask the infrastructure team (the rollback playbook is documented internally).
- Not sure: Stop and ping the team before proceeding. Don't force a failed run forward.
Quick Reference
# Confirm all minions are up
sudo salt '*' test.ping
# Dry run — see what would change, no modifications made
sudo salt-run state.orch orch.patching test=True
# Run the patch
sudo salt-run state.orch orch.patching
# Check which hosts need a reboot after patching
sudo salt '*' cmd.run 'test -f /var/run/reboot-required && echo REBOOT_NEEDED || echo ok'