Archive-Documents/charles-patching-guide.md

TopNotch Gaming — Patching Guide for Charles Tepper

Version Numer 1.0.0 Release Last updated: 2026-05-23 Applies to: All Salt-managed Linux hosts (15 minions)
Run from: undyne (192.168.16.9) — Salt Master

---

Overview

Patching is handled by Salt and runs in three tiers: dev → staging → prod. The orchestrator waits for each tier to succeed before moving to the next, so a bad update won't automatically roll forward to production.

Reboots are disabled by default (controlled via pillar). The patch run upgrades packages only; reboots are a separate decision made after reviewing what changed.

---

Before You Start

SSH into undyne using your AD credentials:

ssh charles@undyne.topnotch.gaming

Confirm Salt is healthy — all minions should return True:

sudo salt '*' test.ping

If any minion doesn't respond, investigate before patching. A minion that's down won't be patched and won't block the run, but you want to know about it.

---

Step 1 — Dry Run (Always Do This First)

sudo salt-run state.orch orch.patching test=True

This shows exactly what would be upgraded on every host without making any changes. Review the output for anything unexpected — kernel updates, Salt itself, or Proxmox packages should not appear (they're held). If they do, stop and flag it.

---

Step 2 — Run the Patch

Once you're happy with the dry run output:

sudo salt-run state.orch orch.patching

The orchestrator will work through tiers sequentially. Each tier must complete cleanly before the next begins. Total runtime varies — expect 5–15 minutes for a full fleet run.

---

Step 3 — Review Output

Look for any False results in the summary. A state returning False means a package update failed on that host. Note the hostname and package, then investigate directly on that host.

---

Reboot Policy

Reboots are not automatic. After patching, check if any hosts need a reboot (e.g. kernel update applied):

sudo salt '*' cmd.run 'test -f /var/run/reboot-required && echo REBOOT_NEEDED || echo ok'

If reboots are needed, coordinate timing and reboot hosts one at a time — especially for the Proxmox cluster nodes (pve, proxVM1, ProxWeb0). Never reboot more than one cluster node simultaneously or quorum is lost and the remaining nodes go read-only.

Standalone node proxVM2 can be rebooted independently without cluster impact.

---

What's Held (Won't Be Patched)

The following packages are pinned and will never be upgraded by the patching run:

• Proxmox stack: pve-manager, proxmox-ve, proxmox-default-kernel, proxmox-kernel-6.14, proxmox-kernel-7.0
• Salt suite: salt-api, salt-cloud, salt-common, salt-master, salt-minion, salt-ssh, salt-syndic

Salt and Proxmox upgrades are handled separately and deliberately — not through the automated patching run.

---

Hosts in Scope

The orchestrator runs all tiers automatically in sequence. The following hosts are Salt-managed and will be patched:

almaelma, granberia, undyne, leviathan, salamander, gnome, sylph, vaultwarden, cosmo, midgar, pve, proxVM1, proxVM2, ProxWeb0, junon, `gw5

Not patched by Salt: npmp-topnotch (Alpine, unmanaged), shiva (bootstrap failing), Windows hosts (luca, chocobo, WINSVR03).

---

If Something Goes Wrong

• Minion stops responding mid-run: SSH directly to the host and check sudo systemctl status salt-minion. Restart it with sudo systemctl restart salt-minion, then re-run.
• Salt itself was accidentally upgraded: Follow the Salt 3007.x rollback procedure — ask the infrastructure team (the rollback playbook is documented internally).
• Not sure: Stop and ping the team before proceeding. Don't force a failed run forward.

---

Quick Reference

# Confirm all minions are up
sudo salt '*' test.ping

# Dry run — see what would change, no modifications made
sudo salt-run state.orch orch.patching test=True

# Run the patch
sudo salt-run state.orch orch.patching

# Check which hosts need a reboot after patching
sudo salt '*' cmd.run 'test -f /var/run/reboot-required && echo REBOOT_NEEDED || echo ok'