Archive-Documents/charles-patching-guide.md

TopNotch Gaming — Patching Guide for Charles Tepper

Version Numer 1.0.0 Release Last updated: 2026-05-23 Applies to: All Salt-managed Linux hosts (15 minions)
Run from: undyne (192.168.16.9) — Salt Master

---

Overview

Patching is handled by Salt and runs in three tiers: dev → staging → prod. The orchestrator waits for each tier to succeed before moving to the next, so a bad update won't automatically roll forward to production.

Reboots are disabled by default (controlled via pillar). The patch run upgrades packages only; reboots are a separate decision made after reviewing what changed.

---

Before You Start

SSH into undyne using your AD credentials:

ssh charles@undyne.topnotch.gaming

Confirm Salt is healthy — all minions should return True:

sudo salt '*' test.ping

If any minion doesn't respond, investigate before patching. A minion that's down won't be patched and won't block the run, but you want to know about it.

---

Step 1 — Dry Run (Always Do This First)

sudo salt-run state.orch orch.patching test=True

This shows exactly what would be upgraded on every host without making any changes. Review the output for anything unexpected — kernel updates, Salt itself, or Proxmox packages should not appear (they're held). If they do, stop and flag it.

---

Step 2 — Run the Patch

Once you're happy with the dry run output:

sudo salt-run state.orch orch.patching

The orchestrator will work through tiers sequentially. Each tier must complete cleanly before the next begins. Total runtime varies — expect 5–15 minutes for a full fleet run.

---

Step 3 — Review Output

Look for any False results in the summary. A state returning False means a package update failed on that host. Note the hostname and package, then investigate directly on that host.

---

Reboot Policy

Reboots are not automatic. After patching, check if any hosts need a reboot (e.g. kernel update applied):

sudo salt '*' cmd.run 'test -f /var/run/reboot-required && echo REBOOT_NEEDED || echo ok'

If reboots are needed, coordinate timing and reboot hosts one at a time — especially for the Proxmox cluster nodes (pve, proxVM1, ProxWeb0). Never reboot more than one cluster node simultaneously or quorum is lost and the remaining nodes go read-only.

Standalone node proxVM2 can be rebooted independently without cluster impact.

---

What's Held (Won't Be Patched)

The following packages are pinned and will never be upgraded by the patching run:

• Proxmox stack: pve-manager, proxmox-ve, proxmox-default-kernel, proxmox-kernel-6.14, proxmox-kernel-7.0
• Salt suite: salt-api, salt-cloud, salt-common, salt-master, salt-minion, salt-ssh, salt-syndic

Salt and Proxmox upgrades are handled separately and deliberately — not through the automated patching run.

---

Hosts in Scope

The orchestrator runs all tiers automatically in sequence. The following hosts are Salt-managed and will be patched:

almaelma, granberia, undyne, leviathan, salamander, gnome, sylph, vaultwarden, cosmo, midgar, pve, proxVM1, proxVM2, ProxWeb0, junon, `gw5

Not patched by Salt: npmp-topnotch (Alpine, unmanaged), shiva (bootstrap failing), Windows hosts (luca, chocobo, WINSVR03).

---

Special Notes about patch targets

leviathan, salamander, and gnome are all AMP Instance Manager Production targets and are active production pieces.

The aim is not to reboot these unless ABSOLUTELY necessary, which after about a month, they -should- be rebooted anyway. If it's that time of the month (har har), notify adupree.

---

If Something Goes Wrong

• Minion stops responding mid-run: SSH directly to the host and check sudo systemctl status salt-minion. Restart it with sudo systemctl restart salt-minion, then re-run.
• Salt itself was accidentally upgraded: Follow the Salt 3007.x rollback procedure — ask the infrastructure team (the rollback playbook is documented internally).
• Not sure: Stop and ping the team before proceeding. Don't force a failed run forward.

---

Quick Reference

# Confirm all minions are up
sudo salt '*' test.ping

# Dry run — see what would change, no modifications made
sudo salt-run state.orch orch.patching test=True

# Run the patch
sudo salt-run state.orch orch.patching

# Check which hosts need a reboot after patching
sudo salt '*' cmd.run 'test -f /var/run/reboot-required && echo REBOOT_NEEDED || echo ok'