155 lines
5.5 KiB
Markdown
155 lines
5.5 KiB
Markdown
# TopNotch Gaming — Patching Guide for Charles Tepper
|
||
|
||
**Version Number:** 1.0.3 Release
|
||
|
||
**Last updated:** 2026-05-26
|
||
|
||
**Applies to:** All Salt-managed Linux hosts (18 minions)
|
||
|
||
**Run from:** undyne (192.168.16.9) — Salt Master
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
Patching is handled by Salt and runs in five tiers: **dev → staging → prod → infra → master**. The orchestrator waits for each tier to succeed before moving to the next, so a bad update won't automatically roll forward.
|
||
|
||
Reboots are **disabled by default** (controlled via pillar). The patch run upgrades packages only; reboots are a separate decision made after reviewing what changed.
|
||
|
||
**Critical:** The `master` tier (undyne) patches last — only after all other tiers succeed. This ensures the Salt master is never taken down mid-rollout. If any tier fails, the rollout halts and undyne is never touched.
|
||
|
||
---
|
||
|
||
## Overview of Machines We Manage Using Salt
|
||
|
||
**INFRA:** Granberia, Undyne, Midgar, Junon, gw5, Kalm, almaelma, pve, proxvm1, proxweb0, proxVM2, cosmo, sylph
|
||
|
||
**PROD:** Leviathan, Gnome, Salamander, vaultwarden, shiva
|
||
|
||
*DO NOTE: PVE, ProxVM1, ProxWeb0, and ProxVM2 are all Proxmox Hypervisors. Under NO circumstance should these ever have any updates that are pertinent to or require a reboot, unless strictly specified under the header "What's Held (Won't Be Patched)".*
|
||
|
||
INFRA should be rebooted every Sunday **if a reboot is required**. Patches should be applied every Sunday.
|
||
|
||
PROD should be rebooted every **first Sunday** regardless of whether a reboot is required or not. Patches should be applied every Sunday.
|
||
|
||
---
|
||
|
||
## Before You Start
|
||
|
||
SSH into undyne using your AD credentials:
|
||
|
||
```bash
|
||
ssh charles@undyne.topnotch.gaming
|
||
```
|
||
|
||
Confirm Salt is healthy — all minions should return `True`:
|
||
|
||
```bash
|
||
sudo salt '*' test.ping
|
||
```
|
||
|
||
If any minion doesn't respond, investigate before patching. A minion that's down won't be patched and won't block the run, but you want to know about it.
|
||
|
||
---
|
||
|
||
## Step 1 — Dry Run (Always Do This First)
|
||
|
||
```bash
|
||
sudo salt-run state.orch orch.patching test=True
|
||
```
|
||
|
||
This shows exactly what would be upgraded on every host without making any changes. Review the output for anything unexpected — kernel updates, Salt itself, or Proxmox packages should **not** appear (they're held). If they do, stop and flag it.
|
||
|
||
---
|
||
|
||
## Step 2 — Run the Patch
|
||
|
||
Once you're happy with the dry run output:
|
||
|
||
```bash
|
||
sudo salt-run state.orch orch.patching
|
||
```
|
||
|
||
The orchestrator will work through tiers sequentially. Each tier must complete cleanly before the next begins. Total runtime varies — expect 5–15 minutes for a full fleet run.
|
||
|
||
---
|
||
|
||
## Step 3 — Review Output
|
||
|
||
Look for any `False` results in the summary. A state returning `False` means a package update failed on that host. Note the hostname and package, then investigate directly on that host.
|
||
|
||
---
|
||
|
||
## Reboot Policy
|
||
|
||
Reboots are **not automatic**. After patching, check if any hosts need a reboot (e.g. kernel update applied):
|
||
|
||
```bash
|
||
sudo salt '*' cmd.run 'test -f /var/run/reboot-required && echo REBOOT_NEEDED || echo ok'
|
||
```
|
||
|
||
If reboots are needed, coordinate timing and reboot hosts one at a time — especially for the Proxmox cluster nodes (pve, proxVM1, ProxWeb0). **Never reboot more than one cluster node simultaneously** or quorum is lost and the remaining nodes go read-only.
|
||
|
||
Standalone node proxVM2 can be rebooted independently without cluster impact.
|
||
|
||
---
|
||
|
||
## What's Held (Won't Be Patched)
|
||
|
||
The following packages are pinned and will never be upgraded by the patching run:
|
||
|
||
- **Proxmox stack:** `pve-manager`, `proxmox-ve`, `proxmox-default-kernel`, `proxmox-kernel-6.14`, `proxmox-kernel-7.0`
|
||
- **Salt suite:** `salt-api`, `salt-cloud`, `salt-common`, `salt-master`, `salt-minion`, `salt-ssh`, `salt-syndic`
|
||
|
||
Salt and Proxmox upgrades are handled separately and deliberately — not through the automated patching run.
|
||
|
||
---
|
||
|
||
## Hosts in Scope
|
||
|
||
The orchestrator runs all tiers automatically in sequence:
|
||
|
||
| Tier | Hosts |
|
||
|------|-------|
|
||
| dev | `almaelma`, `sylph`, `gw5` |
|
||
| staging | `pve`, `proxVM1`, `proxVM2`, `ProxWeb0` |
|
||
| prod | `leviathan`, `gnome`, `salamander`, `vaultwarden`, `kalm`, `shiva` |
|
||
| infra | `cosmo`, `midgar`, `junon`, `granberia` |
|
||
| master | `undyne` |
|
||
|
||
**Not patched by Salt:** `npmp-topnotch` (Alpine, unmanaged), Windows hosts (luca, chocobo, WINSVR03).
|
||
|
||
---
|
||
|
||
## Special Notes About Patch Targets
|
||
|
||
`leviathan`, `salamander`, and `gnome` are all AMP Instance Manager Production targets and are active production pieces.
|
||
|
||
The aim is not to reboot these unless ABSOLUTELY necessary. After about a month of uptime, they *should* be rebooted regardless. If it's that time of the month (har har), notify `adupree`.
|
||
|
||
---
|
||
|
||
## If Something Goes Wrong
|
||
|
||
- **Minion stops responding mid-run:** SSH directly to the host and check `sudo systemctl status salt-minion`. Restart it with `sudo systemctl restart salt-minion`, then re-run.
|
||
- **Salt itself was accidentally upgraded:** Follow the Salt 3007.x rollback procedure — ask the infrastructure team (the rollback playbook is documented internally).
|
||
- **Not sure:** Stop and ping the team before proceeding. Don't force a failed run forward.
|
||
|
||
---
|
||
|
||
## Quick Reference
|
||
|
||
```bash
|
||
# Confirm all minions are up
|
||
sudo salt '*' test.ping
|
||
|
||
# Dry run — see what would change, no modifications made
|
||
sudo salt-run state.orch orch.patching test=True
|
||
|
||
# Run the patch
|
||
sudo salt-run state.orch orch.patching
|
||
|
||
# Check which hosts need a reboot after patching
|
||
sudo salt '*' cmd.run 'test -f /var/run/reboot-required && echo REBOOT_NEEDED || echo ok'
|
||
```
|