Charles
a69114cdaf
First dedicated maintenance instructions document to current server setup (05-23-2026).
122 lines
4.2 KiB
Markdown
122 lines
4.2 KiB
Markdown
# TopNotch Gaming — Patching Guide for Charles Tepper
|
||
|
||
**Version Numer** 1.0.0 Release
|
||
**Last updated:** 2026-05-23
|
||
**Applies to:** All Salt-managed Linux hosts (15 minions)
|
||
**Run from:** undyne (192.168.16.9) — Salt Master
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
Patching is handled by Salt and runs in three tiers: **dev → staging → prod**. The orchestrator waits for each tier to succeed before moving to the next, so a bad update won't automatically roll forward to production.
|
||
|
||
Reboots are **disabled by default** (controlled via pillar). The patch run upgrades packages only; reboots are a separate decision made after reviewing what changed.
|
||
|
||
---
|
||
|
||
## Before You Start
|
||
|
||
SSH into undyne using your AD credentials:
|
||
|
||
```bash
|
||
ssh charles@undyne.topnotch.gaming
|
||
```
|
||
|
||
Confirm Salt is healthy — all minions should return `True`:
|
||
|
||
```bash
|
||
sudo salt '*' test.ping
|
||
```
|
||
|
||
If any minion doesn't respond, investigate before patching. A minion that's down won't be patched and won't block the run, but you want to know about it.
|
||
|
||
---
|
||
|
||
## Step 1 — Dry Run (Always Do This First)
|
||
|
||
```bash
|
||
sudo salt-run state.orch orch.patching test=True
|
||
```
|
||
|
||
This shows exactly what would be upgraded on every host without making any changes. Review the output for anything unexpected — kernel updates, Salt itself, or Proxmox packages should **not** appear (they're held). If they do, stop and flag it.
|
||
|
||
---
|
||
|
||
## Step 2 — Run the Patch
|
||
|
||
Once you're happy with the dry run output:
|
||
|
||
```bash
|
||
sudo salt-run state.orch orch.patching
|
||
```
|
||
|
||
The orchestrator will work through tiers sequentially. Each tier must complete cleanly before the next begins. Total runtime varies — expect 5–15 minutes for a full fleet run.
|
||
|
||
---
|
||
|
||
## Step 3 — Review Output
|
||
|
||
Look for any `False` results in the summary. A state returning `False` means a package update failed on that host. Note the hostname and package, then investigate directly on that host.
|
||
|
||
---
|
||
|
||
## Reboot Policy
|
||
|
||
Reboots are **not automatic**. After patching, check if any hosts need a reboot (e.g. kernel update applied):
|
||
|
||
```bash
|
||
sudo salt '*' cmd.run 'test -f /var/run/reboot-required && echo REBOOT_NEEDED || echo ok'
|
||
```
|
||
|
||
If reboots are needed, coordinate timing and reboot hosts one at a time — especially for the Proxmox cluster nodes (pve, proxVM1, ProxWeb0). **Never reboot more than one cluster node simultaneously** or quorum is lost and the remaining nodes go read-only.
|
||
|
||
Standalone node proxVM2 can be rebooted independently without cluster impact.
|
||
|
||
---
|
||
|
||
## What's Held (Won't Be Patched)
|
||
|
||
The following packages are pinned and will never be upgraded by the patching run:
|
||
|
||
- **Proxmox stack:** `pve-manager`, `proxmox-ve`, `proxmox-default-kernel`, `proxmox-kernel-6.14`, `proxmox-kernel-7.0`
|
||
- **Salt suite:** `salt-api`, `salt-cloud`, `salt-common`, `salt-master`, `salt-minion`, `salt-ssh`, `salt-syndic`
|
||
|
||
Salt and Proxmox upgrades are handled separately and deliberately — not through the automated patching run.
|
||
|
||
---
|
||
|
||
## Hosts in Scope
|
||
|
||
The orchestrator runs all tiers automatically in sequence. The following hosts are Salt-managed and will be patched:
|
||
|
||
`almaelma`, `granberia`, `undyne`, `leviathan`, `salamander`, `gnome`, `sylph`, `vaultwarden`, `cosmo`, `midgar`, `pve`, `proxVM1`, `proxVM2`, `ProxWeb0`, `junon`, `gw5
|
||
|
||
**Not patched by Salt:** `npmp-topnotch` (Alpine, unmanaged), `shiva` (bootstrap failing), Windows hosts (luca, chocobo, WINSVR03).
|
||
|
||
---
|
||
|
||
## If Something Goes Wrong
|
||
|
||
- **Minion stops responding mid-run:** SSH directly to the host and check `sudo systemctl status salt-minion`. Restart it with `sudo systemctl restart salt-minion`, then re-run.
|
||
- **Salt itself was accidentally upgraded:** Follow the Salt 3007.x rollback procedure — ask the infrastructure team (the rollback playbook is documented internally).
|
||
- **Not sure:** Stop and ping the team before proceeding. Don't force a failed run forward.
|
||
|
||
---
|
||
|
||
## Quick Reference
|
||
|
||
```bash
|
||
# Confirm all minions are up
|
||
sudo salt '*' test.ping
|
||
|
||
# Dry run — see what would change, no modifications made
|
||
sudo salt-run state.orch orch.patching test=True
|
||
|
||
# Run the patch
|
||
sudo salt-run state.orch orch.patching
|
||
|
||
# Check which hosts need a reboot after patching
|
||
sudo salt '*' cmd.run 'test -f /var/run/reboot-required && echo REBOOT_NEEDED || echo ok'
|
||
```
|