feat(zec2_rotatekeys): rotate/reset server-side secrets without exposing values

New tool for the leaked/overwritten prod .env case: -Rotate KEY regenerates a
key ON THE SERVER (openssl rand -hex 32) so the value never leaves the box;
-Set KEY takes an operator-known value from a masked prompt and streams it over
SSH stdin (never a command arg, never echoed). Backs the server .env up to a
timestamped .bak first, updates keys atomically (match-or-append), auto-detects
backend/.env from deploy.preserve, restarts only with -Restart, and -WhatIf
previews the plan. Docs added to README + CHANGELOG.
This commit is contained in:
KellyMichels 2026-07-24 12:49:09 -05:00
parent 4ebde3ebcf
commit 4452f54404
4 changed files with 315 additions and 0 deletions

View File

@ -22,6 +22,16 @@ Notable changes to the Evomedia.net Token Savers.
credentials and RSA signing keys. credentials and RSA signing keys.
### Added ### Added
- **`zec2_rotatekeys` — safely rotate/reset server-side secrets** — a new
tool for when a secret leaks or a deploy overwrites a production `.env`
with dev values. `-Rotate KEY` regenerates a key **on the server**
(`openssl rand -hex 32`) so the new value never leaves the box; `-Set KEY`
takes an operator-known value (e.g. `DATABASE_URL`, `ADMIN_EMAIL`) from a
masked prompt and streams it over SSH stdin — never a command argument,
never echoed. Backs the server `.env` up to a timestamped `.bak` first,
updates the key atomically (matches or appends), auto-detects
`backend/.env` from `deploy.preserve`, restarts only with `-Restart`, and
`-WhatIf` previews the plan without touching anything.
- **`zdeploy` server-side health verification (`verify` block)** — projects - **`zdeploy` server-side health verification (`verify` block)** — projects
not published through the edge proxy can declare not published through the edge proxy can declare
`"verify": { "port": ..., "path": "/health", "expect": "..." }` and the `"verify": { "port": ..., "path": "/health", "expect": "..." }` and the

View File

@ -122,6 +122,7 @@ The `.cmd` wrappers are the everyday interface. Every command takes one or more
| `zec2 [<key> ...]` | Quick reachability check (TCP + HTTP + live build version) | | `zec2 [<key> ...]` | Quick reachability check (TCP + HTTP + live build version) |
| `zec2online [<key> ...]` | Deep health check; auto-starts downed stacks, streams diagnostics | | `zec2online [<key> ...]` | Deep health check; auto-starts downed stacks, streams diagnostics |
| `zrepair <key> ...` | Audit + repair compose/proxy state on the server | | `zrepair <key> ...` | Audit + repair compose/proxy state on the server |
| `zec2_rotatekeys <key>` | Rotate/reset secret keys in a project's server-side `.env` (values generated server-side; never printed) |
| `zbackup <key> ... \| all` | Zip local project sources (+ DB dump) to the backups folder | | `zbackup <key> ... \| all` | Zip local project sources (+ DB dump) to the backups folder |
| `zbackup_ec2 [<key> ...]` | Pull DB dumps + server-side data files down from the server | | `zbackup_ec2 [<key> ...]` | Pull DB dumps + server-side data files down from the server |
| `zsync [<key>]` | Copy new backups offsite (or build + mirror a vite dist) | | `zsync [<key>]` | Copy new backups offsite (or build + mirror a vite dist) |
@ -221,6 +222,22 @@ zstop <project> [<project> ...]
`docker compose down` for the selected stacks on the server. Data volumes are preserved; `zdeploy <project>` brings a stack back. (PowerShell script only, no `.cmd` wrapper.) `docker compose down` for the selected stacks on the server. Data volumes are preserved; `zdeploy <project>` brings a stack back. (PowerShell script only, no `.cmd` wrapper.)
#### `zec2_rotatekeys` — rotate server-side secrets
```
zec2_rotatekeys <project> [-Rotate KEY,KEY] [-Set KEY,KEY] [-EnvFile rel/path] [-Restart] [-WhatIf]
```
For when a secret leaks or a deploy overwrites a production `.env` with dev values: rotate or reset keys in a project's **server-side** `.env` without the values ever passing through this machine's shell history, a command argument, or your screen. `-Rotate` keys are regenerated **on the server** with `openssl rand -hex 32` — the new value is written straight into the `.env` there and never leaves the box. `-Set` keys are typed into a masked prompt and streamed to the server over SSH stdin (never a command argument, never echoed), for operator-known values like `DATABASE_URL` or `ADMIN_EMAIL`. The current server `.env` is copied to a timestamped `.bak` before any change; the KEY line is updated atomically, matching an existing key or appending it. The env file is auto-detected from the project's `deploy.preserve` (first `*.env`) or defaults to `.env` — override with `-EnvFile backend/.env`. Nothing restarts unless you pass `-Restart`. Being high-impact, it confirms before writing; `-WhatIf` prints the exact plan and changes nothing.
```powershell
# Preview only — see exactly what would change, change nothing:
zec2_rotatekeys pyapp -Rotate JWT_SECRET -Set DATABASE_URL,ADMIN_EMAIL -WhatIf
# Regenerate the JWT secret, restore the operator-known values, then restart:
zec2_rotatekeys pyapp -Rotate JWT_SECRET -Set DATABASE_URL,ADMIN_EMAIL,ADMIN_PASSWORD -Restart
```
### Backups ### Backups
#### `zbackup` — local backups #### `zbackup` — local backups

6
zec2_rotatekeys.cmd Normal file
View File

@ -0,0 +1,6 @@
REM Evomedia.net Token Savers — https://github.com/kellymichels/zscripts-token-savers
REM Created by Kelly Michels · dev@evomedia.net
REM Licensed under the MIT License. See LICENSE.
@echo off
powershell -NoProfile -ExecutionPolicy Bypass -File "%~dp0zec2_rotatekeys.ps1" %*

282
zec2_rotatekeys.ps1 Normal file
View File

@ -0,0 +1,282 @@
# Evomedia.net Token Savers — https://github.com/kellymichels/zscripts-token-savers
# Created by Kelly Michels · dev@evomedia.net
# Licensed under the MIT License. See LICENSE.
# zec2_rotatekeys.ps1 - rotate / reset secret keys in a project's SERVER-SIDE
# .env, in place, without the values ever passing through this machine's shell
# history, command args, or the operator's screen.
#
# Why this exists: if a deploy ever ships a dev .env over a project's production
# .env (or a secret leaks), you need to (1) regenerate the machine secrets and
# (2) restore the correct operator-known values on the server - safely.
#
# How it stays safe:
# * -Rotate keys are regenerated ON THE SERVER (openssl rand -hex 32). The new
# value is created on the box and written straight into the .env there; it is
# never sent from here, never printed.
# * -Set keys are typed into a masked prompt and streamed to the server over
# SSH stdin (the encrypted channel) - never placed in a command argument
# (where `ps`/history would capture it) and never echoed back.
# * The current server .env is copied to a timestamped .bak before any change.
# * -WhatIf prints the exact plan and touches nothing. High-impact, so it
# confirms before writing unless you pass -Confirm:$false.
# * It does NOT restart the app unless you pass -Restart (prod restarts are a
# deliberate, separate decision).
#
# Usage:
# zec2_rotatekeys <project> [-Rotate K1,K2] [-Set K1,K2] [-EnvFile rel/path]
# [-Restart] [-HostName ip] [-WhatIf] [-Confirm:$false]
#
# Examples:
# # Preview only - see exactly what would change, change nothing:
# zec2_rotatekeys pyapp -Rotate JWT_SECRET -Set DATABASE_URL,ADMIN_EMAIL -WhatIf
#
# # Regenerate the JWT secret and restore the operator-known values, then
# # restart the app so it picks them up:
# zec2_rotatekeys pyapp -Rotate JWT_SECRET -Set DATABASE_URL,ADMIN_EMAIL,ADMIN_PASSWORD -Restart
#
# The env file is auto-detected from the project's deploy.preserve (first *.env
# entry) or defaults to ".env"; override with -EnvFile (relative to remote.path,
# e.g. -EnvFile backend/.env).
[CmdletBinding(SupportsShouldProcess = $true, ConfirmImpact = 'High')]
param(
[Parameter(Position = 0)][string]$Project,
[string[]]$Rotate = @(),
[string[]]$Set = @(),
[string]$EnvFile,
[switch]$Restart,
[string]$HostName
)
$ErrorActionPreference = "Stop"
. (Join-Path $PSScriptRoot "ZHelpers.ps1")
Start-ZTracking
function Show-Usage {
Write-Host ""
Write-Host "Usage: zec2_rotatekeys <project> [-Rotate K1,K2] [-Set K1,K2] [-EnvFile rel/path] [-Restart] [-WhatIf]" -ForegroundColor Yellow
Write-Host " -Rotate keys regenerated on the server with a fresh random secret (openssl rand -hex 32)" -ForegroundColor Gray
Write-Host " -Set keys set from a masked prompt, streamed over SSH stdin (for operator-known values)" -ForegroundColor Gray
try { $keys = (Get-ZProjectKeys) -join ', '; Write-Host " Projects: $keys" -ForegroundColor Gray } catch { }
Write-Host ""
Write-Host " Example: zec2_rotatekeys pyapp -Rotate JWT_SECRET -Set DATABASE_URL,ADMIN_EMAIL,ADMIN_PASSWORD -Restart" -ForegroundColor DarkGray
}
# Server-side worker. Static (no secrets): -Rotate generates its value here on
# the box; -Set reads it from stdin. Uploaded base64-encoded so line endings and
# quoting survive the trip intact. Updates the KEY line atomically via python.
$rkHelper = @'
#!/usr/bin/env bash
set -uo pipefail
env_file="${1:-}"; key="${2:-}"; mode="${3:-}"
if [ -z "$env_file" ] || [ -z "$key" ] || [ -z "$mode" ]; then echo "BAD_ARGS"; exit 5; fi
if [ ! -f "$env_file" ]; then echo "ENV_MISSING:$env_file"; exit 2; fi
case "$mode" in
rotate)
command -v openssl >/dev/null 2>&1 || { echo "NO_OPENSSL"; exit 6; }
val="$(openssl rand -hex 32)"
;;
set)
IFS= read -r val || true
val="${val%$'\r'}"
if [ -z "$val" ]; then echo "EMPTY_VALUE:$key"; exit 4; fi
;;
*) echo "BAD_MODE:$mode"; exit 3 ;;
esac
KEY="$key" VAL="$val" python3 - "$env_file" <<'PY'
import os, sys, tempfile
path = sys.argv[1]
k = os.environ['KEY']; v = os.environ['VAL']
with open(path, 'r') as fh:
lines = fh.read().splitlines()
out = []
found = False
for ln in lines:
s = ln.lstrip()
if (not s.startswith('#')) and ('=' in ln) and (ln.split('=', 1)[0].strip() == k):
out.append(k + '=' + v)
found = True
else:
out.append(ln)
if not found:
out.append(k + '=' + v)
d = os.path.dirname(path) or '.'
fd, tmp = tempfile.mkstemp(dir=d)
try:
with os.fdopen(fd, 'w') as fh:
fh.write('\n'.join(out) + '\n')
os.chmod(tmp, 0o600)
os.replace(tmp, path)
except Exception:
try:
os.unlink(tmp)
except OSError:
pass
raise
print('OK:' + k)
PY
'@
try {
if (-not $Project) { Show-Usage; Stop-ZTracking; exit 1 }
$Rotate = @($Rotate | Where-Object { $_ -and $_.Trim() } | ForEach-Object { $_.Trim() })
$Set = @($Set | Where-Object { $_ -and $_.Trim() } | ForEach-Object { $_.Trim() })
if ($Rotate.Count -eq 0 -and $Set.Count -eq 0) {
Write-Host "ERROR: nothing to do - pass -Rotate and/or -Set with at least one key." -ForegroundColor Red
Show-Usage; Stop-ZTracking; exit 1
}
# A key can't be both regenerated and set - -Set (explicit value) wins.
$overlap = @($Rotate | Where-Object { $Set -contains $_ })
if ($overlap.Count -gt 0) {
Write-Host "NOTE: $($overlap -join ', ') given to both -Rotate and -Set; using -Set (explicit value)." -ForegroundColor Yellow
$Rotate = @($Rotate | Where-Object { $Set -notcontains $_ })
}
$cfg = Get-ZConfig
$proj = Get-ZProject -Key $Project # exits with a clear error on a bad key
$remotePath = $proj.remote.path
if (-not $remotePath) {
Write-Host "ERROR: project '$Project' has no remote.path in zconfig.json - nothing to rotate on the server." -ForegroundColor Red
Stop-ZTracking; exit 1
}
# Env file location: -EnvFile wins; else first *.env in deploy.preserve; else ".env".
if (-not $EnvFile) {
$EnvFile = ".env"
if ($proj.deploy -and $proj.deploy.preserve) {
$cand = @($proj.deploy.preserve | Where-Object { $_ -match '\.env$' }) | Select-Object -First 1
if ($cand) { $EnvFile = $cand }
}
}
$EnvFile = $EnvFile -replace '\\', '/'
$remoteEnv = "$remotePath/$EnvFile"
$ip = if ($HostName) { $HostName } else { $cfg.ec2.ip }
$pem = $cfg.ec2.pemKey
$target = "$($cfg.ec2.user)@$ip"
$sshOpts = @('-o', 'StrictHostKeyChecking=no', '-o', 'ConnectTimeout=15', '-i', $pem)
$remoteHelper = "/tmp/zrk_$PID.sh"
Write-Host ""
Write-Host "=== zec2_rotatekeys ($($proj.label)) ===" -ForegroundColor Cyan
Write-Host " Server: $target" -ForegroundColor DarkGray
Write-Host " Env file: $remoteEnv" -ForegroundColor DarkGray
if ($Rotate.Count) { Write-Host " Rotate (fresh random, server-side): $($Rotate -join ', ')" -ForegroundColor Gray }
if ($Set.Count) { Write-Host " Set (masked prompt, streamed): $($Set -join ', ')" -ForegroundColor Gray }
if ($Restart) { Write-Host " Then: restart the app container" -ForegroundColor Gray }
Write-Host ""
$action = @()
if ($Rotate.Count) { $action += "rotate [$($Rotate -join ',')]" }
if ($Set.Count) { $action += "set [$($Set -join ',')]" }
if ($Restart) { $action += "restart" }
if (-not $PSCmdlet.ShouldProcess("${target}:$remoteEnv", ($action -join ' + '))) {
Write-Host "Preview only - no changes made." -ForegroundColor Yellow
Stop-ZTracking; exit 0
}
# --- confirm the env file actually exists before we touch anything --------
$exists = (ssh @sshOpts $target "test -f $remoteEnv && echo EXISTS || echo MISSING") | Select-Object -Last 1
if ($LASTEXITCODE -ne 0) { throw "Could not reach $target over SSH (exit $LASTEXITCODE)." }
if ($exists -ne "EXISTS") {
throw "Env file not found on the server: $remoteEnv. Check remote.path / -EnvFile."
}
# --- back up the current server .env --------------------------------------
$ts = Get-Date -Format "yyyyMMdd-HHmmss"
$backup = "$remoteEnv.bak.$ts"
ssh @sshOpts $target "cp -p $remoteEnv $backup"
if ($LASTEXITCODE -ne 0) { throw "Backup of $remoteEnv failed (exit $LASTEXITCODE) - aborting before any change." }
Write-Host " Backed up server .env -> $backup" -ForegroundColor DarkGray
# --- upload the worker (base64: no CR / quoting surprises) -----------------
$helperLf = $rkHelper -replace "`r`n", "`n"
$b64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($helperLf))
ssh @sshOpts $target "echo $b64 | base64 -d > $remoteHelper && chmod 700 $remoteHelper"
if ($LASTEXITCODE -ne 0) { throw "Failed to stage the rotation helper on the server (exit $LASTEXITCODE)." }
$done = @()
$failed = @()
try {
# --- rotate: value generated on the server, never seen here -----------
foreach ($key in $Rotate) {
$r = (ssh @sshOpts $target "bash $remoteHelper $remoteEnv $key rotate") | Select-Object -Last 1
if ($LASTEXITCODE -eq 0 -and $r -like "OK:*") {
Write-Host " rotated $key" -ForegroundColor Green
$done += "$key (rotated)"
} else {
Write-Host " FAILED $key ($r)" -ForegroundColor Red
$failed += "$key ($r)"
}
}
# --- set: masked prompt -> SSH stdin, never in args or on screen ------
foreach ($key in $Set) {
$secure = Read-Host -Prompt " New value for $key" -AsSecureString
$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secure)
try {
$plain = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr)
} finally {
[Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr)
}
if ([string]::IsNullOrEmpty($plain)) {
Write-Host " skipped $key (no value entered)" -ForegroundColor Yellow
$failed += "$key (empty - skipped)"
$plain = $null
continue
}
$r = ($plain | ssh @sshOpts $target "bash $remoteHelper $remoteEnv $key set") | Select-Object -Last 1
$rc = $LASTEXITCODE
$plain = $null # drop the plaintext from memory promptly
if ($rc -eq 0 -and $r -like "OK:*") {
Write-Host " set $key" -ForegroundColor Green
$done += "$key (set)"
} else {
Write-Host " FAILED $key ($r)" -ForegroundColor Red
$failed += "$key ($r)"
}
}
} finally {
ssh @sshOpts $target "rm -f $remoteHelper" | Out-Null
}
# --- optional app restart -------------------------------------------------
$restarted = $false
if ($Restart -and $done.Count -gt 0) {
$composeDir = if ($proj.remote.composeDir) { $proj.remote.composeDir } else { $remotePath }
$svc = if ($proj.remote.appService) { $proj.remote.appService } else { "app" }
Write-Host ""
Write-Host " Restarting service '$svc' in $composeDir ..." -ForegroundColor Cyan
ssh @sshOpts $target "cd $composeDir && sudo COMPOSE_BAKE=false docker compose restart $svc"
if ($LASTEXITCODE -eq 0) { Write-Host " Restarted $svc." -ForegroundColor Green; $restarted = $true }
else { Write-Host " WARNING: restart of '$svc' failed (exit $LASTEXITCODE) - restart it manually." -ForegroundColor Red }
} elseif ($Restart) {
Write-Host " Skipping restart - no keys were changed." -ForegroundColor Yellow
}
# --- summary --------------------------------------------------------------
Write-Host ""
Write-Host "=== Summary ===" -ForegroundColor Cyan
Write-Host " Changed: $(if ($done.Count) { $done -join ', ' } else { 'none' })" -ForegroundColor $(if ($done.Count) { 'Green' } else { 'Yellow' })
if ($failed.Count) { Write-Host " Failed: $($failed -join ', ')" -ForegroundColor Red }
Write-Host " Backup: $backup (delete once verified)" -ForegroundColor DarkGray
if ($Restart -and -not $restarted -and $done.Count -gt 0) {
Write-Host " Restart: NOT done - restart the app so it loads the new values." -ForegroundColor Yellow
} elseif (-not $Restart -and $done.Count -gt 0) {
Write-Host " Note: the app is still running with the OLD values - restart it (or re-run with -Restart)." -ForegroundColor Yellow
}
Write-Host ""
Stop-ZTracking
if ($failed.Count) { exit 2 }
exit 0
}
catch {
Write-Host "ERROR: $($_.Exception.Message)" -ForegroundColor Red
Stop-ZTracking
exit 1
}