Server Maintenance Instructions

First dedicated maintenance instructions document to current server setup (05-23-2026).

charles-patching-guide.md

@@ -0,0 +1,121 @@
+# TopNotch Gaming — Patching Guide for Charles Tepper
+
+**Version Numer** 1.0.0 Release
+**Last updated:** 2026-05-23
+**Applies to:** All Salt-managed Linux hosts (15 minions)  
+**Run from:** undyne (192.168.16.9) — Salt Master
+
+---
+
+## Overview
+
+Patching is handled by Salt and runs in three tiers: **dev → staging → prod**. The orchestrator waits for each tier to succeed before moving to the next, so a bad update won't automatically roll forward to production.
+
+Reboots are **disabled by default** (controlled via pillar). The patch run upgrades packages only; reboots are a separate decision made after reviewing what changed.
+
+---
+
+## Before You Start
+
+SSH into undyne using your AD credentials:
+
+```bash
+ssh charles@undyne.topnotch.gaming
+```
+
+Confirm Salt is healthy — all minions should return `True`:
+
+```bash
+sudo salt '*' test.ping
+```
+
+If any minion doesn't respond, investigate before patching. A minion that's down won't be patched and won't block the run, but you want to know about it.
+
+---
+
+## Step 1 — Dry Run (Always Do This First)
+
+```bash
+sudo salt-run state.orch orch.patching test=True
+```
+
+This shows exactly what would be upgraded on every host without making any changes. Review the output for anything unexpected — kernel updates, Salt itself, or Proxmox packages should **not** appear (they're held). If they do, stop and flag it.
+
+---
+
+## Step 2 — Run the Patch
+
+Once you're happy with the dry run output:
+
+```bash
+sudo salt-run state.orch orch.patching
+```
+
+The orchestrator will work through tiers sequentially. Each tier must complete cleanly before the next begins. Total runtime varies — expect 5–15 minutes for a full fleet run.
+
+---
+
+## Step 3 — Review Output
+
+Look for any `False` results in the summary. A state returning `False` means a package update failed on that host. Note the hostname and package, then investigate directly on that host.
+
+---
+
+## Reboot Policy
+
+Reboots are **not automatic**. After patching, check if any hosts need a reboot (e.g. kernel update applied):
+
+```bash
+sudo salt '*' cmd.run 'test -f /var/run/reboot-required && echo REBOOT_NEEDED || echo ok'
+```
+
+If reboots are needed, coordinate timing and reboot hosts one at a time — especially for the Proxmox cluster nodes (pve, proxVM1, ProxWeb0). **Never reboot more than one cluster node simultaneously** or quorum is lost and the remaining nodes go read-only.
+
+Standalone node proxVM2 can be rebooted independently without cluster impact.
+
+---
+
+## What's Held (Won't Be Patched)
+
+The following packages are pinned and will never be upgraded by the patching run:
+
+- **Proxmox stack:** `pve-manager`, `proxmox-ve`, `proxmox-default-kernel`, `proxmox-kernel-6.14`, `proxmox-kernel-7.0`
+- **Salt suite:** `salt-api`, `salt-cloud`, `salt-common`, `salt-master`, `salt-minion`, `salt-ssh`, `salt-syndic`
+
+Salt and Proxmox upgrades are handled separately and deliberately — not through the automated patching run.
+
+---
+
+## Hosts in Scope
+
+The orchestrator runs all tiers automatically in sequence. The following hosts are Salt-managed and will be patched:
+
+`almaelma`, `granberia`, `undyne`, `leviathan`, `salamander`, `gnome`, `sylph`, `vaultwarden`, `cosmo`, `midgar`, `pve`, `proxVM1`, `proxVM2`, `ProxWeb0`, `junon`, `gw5
+
+**Not patched by Salt:** `npmp-topnotch` (Alpine, unmanaged), `shiva` (bootstrap failing), Windows hosts (luca, chocobo, WINSVR03).
+
+---
+
+## If Something Goes Wrong
+
+- **Minion stops responding mid-run:** SSH directly to the host and check `sudo systemctl status salt-minion`. Restart it with `sudo systemctl restart salt-minion`, then re-run.
+- **Salt itself was accidentally upgraded:** Follow the Salt 3007.x rollback procedure — ask the infrastructure team (the rollback playbook is documented internally).
+- **Not sure:** Stop and ping the team before proceeding. Don't force a failed run forward.
+
+---
+
+## Quick Reference
+
+```bash
+# Confirm all minions are up
+sudo salt '*' test.ping
+
+# Dry run — see what would change, no modifications made
+sudo salt-run state.orch orch.patching test=True
+
+# Run the patch
+sudo salt-run state.orch orch.patching
+
+# Check which hosts need a reboot after patching
+sudo salt '*' cmd.run 'test -f /var/run/reboot-required && echo REBOOT_NEEDED || echo ok'
+```