# TopNotch Gaming — Patching Guide for Charles Tepper

**Version Number:** 1.0.3 Release

**Last updated:** 2026-05-26

**Applies to:** All Salt-managed Linux hosts (18 minions)

**Run from:** undyne (192.168.16.9) — Salt Master

---

## Overview

Patching is handled by Salt and runs in five tiers: **dev → staging → prod → infra → master**. The orchestrator waits for each tier to succeed before moving to the next, so a bad update won't automatically roll forward.

Reboots are **disabled by default** (controlled via pillar). The patch run upgrades packages only; reboots are a separate decision made after reviewing what changed.

**Critical:** The `master` tier (undyne) patches last — only after all other tiers succeed. This ensures the Salt master is never taken down mid-rollout. If any tier fails, the rollout halts and undyne is never touched.

---

## Overview of Machines We Manage Using Salt

**INFRA:** Granberia, Undyne, Midgar, Junon, gw5, Kalm, almaelma, pve, proxvm1, proxweb0, proxVM2, cosmo, sylph

**PROD:** Leviathan, Gnome, Salamander, vaultwarden, shiva

*DO NOTE: PVE, ProxVM1, ProxWeb0, and ProxVM2 are all Proxmox Hypervisors. Under NO circumstance should these ever have any updates that are pertinent to or require a reboot, unless strictly specified under the header "What's Held (Won't Be Patched)".*

INFRA should be rebooted every Sunday **if a reboot is required**. Patches should be applied every Sunday.

PROD should be rebooted every **first Sunday** regardless of whether a reboot is required or not. Patches should be applied every Sunday.

---

## Before You Start

SSH into undyne using your AD credentials:

```bash
ssh charles@undyne.topnotch.gaming
```

Confirm Salt is healthy — all minions should return `True`:

```bash
sudo salt '*' test.ping
```

If any minion doesn't respond, investigate before patching. A minion that's down won't be patched and won't block the run, but you want to know about it.

---

## Step 1 — Dry Run (Always Do This First)

```bash
sudo salt-run state.orch orch.patching test=True
```

This shows exactly what would be upgraded on every host without making any changes. Review the output for anything unexpected — kernel updates, Salt itself, or Proxmox packages should **not** appear (they're held). If they do, stop and flag it.

---

## Step 2 — Run the Patch

Once you're happy with the dry run output:

```bash
sudo salt-run state.orch orch.patching
```

The orchestrator will work through tiers sequentially. Each tier must complete cleanly before the next begins. Total runtime varies — expect 5–15 minutes for a full fleet run.

---

## Step 3 — Review Output

Look for any `False` results in the summary. A state returning `False` means a package update failed on that host. Note the hostname and package, then investigate directly on that host.

---

## Reboot Policy

Reboots are **not automatic**. After patching, check if any hosts need a reboot (e.g. kernel update applied):

```bash
sudo salt '*' cmd.run 'test -f /var/run/reboot-required && echo REBOOT_NEEDED || echo ok'
```

If reboots are needed, coordinate timing and reboot hosts one at a time — especially for the Proxmox cluster nodes (pve, proxVM1, ProxWeb0). **Never reboot more than one cluster node simultaneously** or quorum is lost and the remaining nodes go read-only.

Standalone node proxVM2 can be rebooted independently without cluster impact.

---

## What's Held (Won't Be Patched)

The following packages are pinned and will never be upgraded by the patching run:

- **Proxmox stack:** `pve-manager`, `proxmox-ve`, `proxmox-default-kernel`, `proxmox-kernel-6.14`, `proxmox-kernel-7.0`
- **Salt suite:** `salt-api`, `salt-cloud`, `salt-common`, `salt-master`, `salt-minion`, `salt-ssh`, `salt-syndic`

Salt and Proxmox upgrades are handled separately and deliberately — not through the automated patching run.

---

## Hosts in Scope

The orchestrator runs all tiers automatically in sequence:

| Tier | Hosts |
|------|-------|
| dev | `almaelma`, `sylph`, `gw5` |
| staging | `pve`, `proxVM1`, `proxVM2`, `ProxWeb0` |
| prod | `leviathan`, `gnome`, `salamander`, `vaultwarden`, `kalm`, `shiva` |
| infra | `cosmo`, `midgar`, `junon`, `granberia` |
| master | `undyne` |

**Not patched by Salt:** `npmp-topnotch` (Alpine, unmanaged), Windows hosts (luca, chocobo, WINSVR03).

---

## Special Notes About Patch Targets

`leviathan`, `salamander`, and `gnome` are all AMP Instance Manager Production targets and are active production pieces.

The aim is not to reboot these unless ABSOLUTELY necessary. After about a month of uptime, they *should* be rebooted regardless. If it's that time of the month (har har), notify `adupree`.

---

## If Something Goes Wrong

- **Minion stops responding mid-run:** SSH directly to the host and check `sudo systemctl status salt-minion`. Restart it with `sudo systemctl restart salt-minion`, then re-run.
- **Salt itself was accidentally upgraded:** Follow the Salt 3007.x rollback procedure — ask the infrastructure team (the rollback playbook is documented internally).
- **Not sure:** Stop and ping the team before proceeding. Don't force a failed run forward.

---

## Quick Reference

```bash
# Confirm all minions are up
sudo salt '*' test.ping

# Dry run — see what would change, no modifications made
sudo salt-run state.orch orch.patching test=True

# Run the patch
sudo salt-run state.orch orch.patching

# Check which hosts need a reboot after patching
sudo salt '*' cmd.run 'test -f /var/run/reboot-required && echo REBOOT_NEEDED || echo ok'
```