feat(zdeploy): sync deploy hardening from the private toolkit (#53)

Nine fixes that had accumulated only in the private copy. Each one is a
production failure that already happened:

  * ssh -n on the deploy path. Without it ssh reads its stdin, inherits
    the console handle under PowerShell, and can block forever. The
    existing timeouts cannot catch it - they bound a connection that is
    dying, and this one was never established. Get-Ec2ScpOpts derives
    from the same list minus -n, because scp rejects it with a usage
    error that reads like an unrelated failure.

  * Invoke-DeployGitPull now switches TO the default branch instead of
    pulling whatever is checked out. Pulling the current branch breaks
    as soon as the remote deletes branches on merge, and worse, could
    ship a feature branch to production. Refuses over local changes,
    excluding the files the deploy itself stamps.

  * nextjs handler resolves composeDir. It ran compose against
    remote.path, so a project whose compose lives in a subdirectory
    recycled whatever docker-compose.yml sat at the project root -
    usually the local dev one shipped in the same archive. Two deploys
    in a row exited 0 having shipped nothing, and verification passed
    because the untouched old container still answered.

  * BUILD BEFORE DOWN. The old order took the site offline for the whole
    build and left it offline if the build failed - one deploy served
    502 for 19 hours with no container running. The old image now keeps
    serving until the new one is ready.

  * compose build takes the named service, so a compose file that does
    not define it fails loudly instead of succeeding with nothing to do.

  * deploy.verifyHost overrides domain for verification only, for when a
    domain is retired ahead of its replacement.

  * deploy.stampCmd writes the build version from git before zipping,
    then reverts the tree so the stamp cannot block the next deploy.

  * Optional ztokens pseudo-project and passive usage records. Both
    degrade to a printed skip when no sibling ztokens checkout exists.

Sanitized on the way across, per the public-repo rule: the war stories
that make these comments worth reading are kept, the private product
names, domains, internal script names, outage dates and host figures are
not. Also dropped a "See issue #28" pointer that resolves to an
unrelated PR in this repo, and two references to scripts that exist only
in the private toolkit.

CHECKSUMS.txt refreshed alongside, per the manifest rule.

Tests: 222/222 (the 4 failures before the refresh were the checksum
suite correctly flagging both edited files).

CHECKSUMS.txt

@@ -8,14 +8,14 @@ b195d2f06601498bb2ce264eb01b6fa567fbbe5396bf1f2e11240c0c7c269ea7  zbackup_ec2.ps
 f443ca5f48537f5d580675ba4ba26d9d06ed1c0db5ec8f7c8a800874f6eed393  zchecksums.cmd
 68ab969da2e2b9d6194a6aab0e27a176d8e67bdc799f32964aa4ac6f232ae3b0  zchecksums.ps1
 92a4405bbee47bbf7ef37e4fcdba3c6b14c63e3acaf494b96663ab20a8e36d75  zdeploy.cmd
-b7a7efab962d89563cb160d43ef48e4356968c9059f1d4ba05ec51ec81c388db  zdeploy.ps1
+677b7f346e5e70a6e4d82a131896179e790be4c6344d51d8a0575ec7e6503ea6  zdeploy.ps1
 0eda8a0a48651940724801d677cc762047a56352c9b9b0323437e3c394d1f253  zec2.cmd
 0a705df645a21d91385a67705c864ba9677a8436678b27dc51b81fff970bb45b  zec2.ps1
 43a5b030db7f5142b339a2b3acc32ebf2c885483d9e1295f09c771aa310ad857  zec2_rotatekeys.cmd
 8c779b5fad01a752611aca7244df5fbf0bc8b9671a7825690dd87c249310f98a  zec2_rotatekeys.ps1
 6984ed519e617c41372f4cfbd393cd3638bd83031f27a7a2643ab47b6dba5b17  zec2online.cmd
 7056287c5fffce3a0360e9a05e40eacdb18ac88f0066f55a9c49e2f3e47a5746  zec2online.ps1
-28556e1c128ba17eac6a9a3c2288658ac19c89396097bc093c48af240418bf2e  ZHelpers.ps1
+5367dfc2ea5776d942673b641d60cf0f7475754a3dd5c62de9b5959010533b67  ZHelpers.ps1
 2d4acc784f1fdd82e9db7cc39bc732158ea9cee981d1b22d1a143160e8a1a632  zkill.cmd
 08c88bfd0f7966b3a4c7df6c45b6e667efedf939e83ced445f05d5f53b800462  zkill.ps1
 bb7d971a2b0113698ecdd7fcaaaa9f3ada9a80d39a984315d9bc6d3e65c6189e  ZKiller.ps1

ZHelpers.ps1

@@ -14,9 +14,9 @@ $script:ArchiveExtensions = @(
 # Directories whose archives are BUILD INPUTS, not incidental bloat, and so are
 # exempt from ArchiveExtensions. A project that vendors a dependency as
 # vendor/*.tgz (common when a bundler cannot resolve `file:` links outside the
-# project root) needs that tarball in the deploy zip — dropping it makes a
+# project root) needs that tarball in the deploy zip - dropping it makes a
-# Dockerfile's `COPY vendor ./vendor` fail at image build time, which is a
+# Dockerfile's `COPY vendor ./vendor` fail at image build, which is a confusing
-# confusing way to discover the archive filter ate a required file.
+# way to discover the archive filter ate a required build input.
 $script:ArchiveKeepDirNames = @('vendor')
 $script:ScriptExtensions = @('.ps1', '.cmd', '.bat')
 $script:JunkExtensions = @(
@@ -140,6 +140,23 @@ function Get-Ec2Home {
 # ConnectTimeout bounds the TCP connect. ServerAlive* bound everything after
 # it, so a session that dies mid-command (dropped VPN, laptop asleep, host
 # rebooting) errors out in about a minute instead of hanging indefinitely.
+# zec2online.ps1 and zsetup_mail.ps1 already set ConnectTimeout; the deploy
+# path, the one place a hang costs the most, set none of them.
+#
+# -n is the one that fixes the hang these options did NOT catch. Without it ssh
+# reads its stdin and forwards it to the remote command, and under PowerShell it
+# inherits the console handle - so it can block forever waiting on input nobody
+# is going to type. The timeouts above cannot help: they bound a connection that
+# is dying, and this one was never established. One deploy stopped under
+# "ensure unzip installed", a step whose body short-circuits when unzip is
+# already present; the server showed no ssh session at all (`who` empty, no
+# docker build running), which is what a client-side stdin block looks like
+# from the other end. The zip had uploaded and prod stayed a release behind.
+#
+# Safe here because nothing that pipes stdin INTO ssh uses these options:
+# zdeploy passes only command strings. Any script that DOES pipe into ssh must
+# build its own option array - adding -n to those would break them, so do not
+# hoist this beyond the deploy path.
 function Get-Ec2SshOpts {
     return @(
         '-n',
@@ -154,7 +171,8 @@ function Get-Ec2SshOpts {
 # The same options for scp, which does NOT accept -n: OpenSSH's scp exits 1 with
 # "unknown option -- n" and prints its usage block. That failure is easy to
 # misread, because the caller's own error text is what the operator sees while
-# the usage text scrolls past above it.
+# the usage text scrolls past above it - one deploy reported "Likely server disk
+# space" on a box with plenty of room.
 #
 # Derived from Get-Ec2SshOpts rather than duplicated, so the timeouts can never
 # drift apart between the two transports.
@@ -191,13 +209,28 @@ function Invoke-Ec2Step {
 
 # ── Deploy git pull ──────────────────────────────────────────────────────────
 
-# Fast-forward the project's checkout before a deploy when deploy.gitPull is set.
+# Put the project's checkout ON the default branch and fast-forward it before
-# zdeploy zips the working tree and does NOT otherwise pull, so after a merged
+# a deploy, when deploy.gitPull is set. zdeploy zips the working tree and does
-# PR the checkout can sit behind origin and the deploy would ship stale code
+# NOT otherwise pull, so after a merged PR the checkout can sit behind origin
-# while still bumping the build number (looks successful, changes nothing).
+# and the deploy would ship stale code while still bumping the build number
-# Aborts the deploy on a failed pull rather than shipping uncertain code. Runs
+# (looks successful, changes nothing).
-# git bare (no 2>&1) and checks $LASTEXITCODE, matching Invoke-Ec2Step under
+#
-# $ErrorActionPreference='Stop'.
+# Deploys ship the default branch, so this SWITCHES to it rather than pulling
+# whatever branch happens to be checked out. The old behaviour pulled the
+# current branch, which breaks as soon as the remote deletes branches on merge:
+# a checkout still sitting on its just-merged PR branch pulls a ref the merge
+# deleted, and the deploy dies on "no such ref was fetched". Worse, when the ref
+# DID still exist, pulling the feature branch meant a deploy could ship a
+# branch rather than the default.
+#
+# The switch refuses to run over local changes: a dirty tree aborts the deploy
+# with the file list rather than risk tangling uncommitted work. The stale
+# branch is left in place for the operator to delete - under squash merges
+# only a content diff can prove it safe, and a deploy is not the place to
+# make that call.
+#
+# Runs git bare (no 2>&1) and checks $LASTEXITCODE, matching Invoke-Ec2Step
+# under $ErrorActionPreference='Stop'.
 function Invoke-DeployGitPull {
     param([Parameter(Mandatory)]$Proj)
     if (-not ($Proj.deploy -and $Proj.deploy.gitPull)) { return }
@@ -206,24 +239,72 @@ function Invoke-DeployGitPull {
         Write-Host "  gitPull set but '$root' is not a git repo - skipping pull." -ForegroundColor Yellow
         return
     }
-    Write-Host "`n--- [0] git sync (fetch + ff-only merge) ---" -ForegroundColor Cyan
+    Write-Host "`n--- [0] git sync default branch ---" -ForegroundColor Cyan
     Push-Location -LiteralPath $root
     try {
-        $branch = (git rev-parse --abbrev-ref HEAD)
+        # Same PS 5.1 trap Invoke-Ec2Step documents: under the deploy's
-        Write-Host "  Branch: $branch" -ForegroundColor DarkGray
+        # ErrorActionPreference='Stop', a stderr REDIRECT on a native command
-        # Fetch explicitly, then fast-forward against the remote-tracking ref -
+        # (the 2>$null on symbolic-ref below) wraps stderr lines in
-        # not `git pull`. Pull merges whatever FETCH_HEAD marks "for merge",
+        # terminating ErrorRecords. Success is judged by $LASTEXITCODE
-        # and a concurrent fetch in the same repo (an editor's background
+        # throughout, so drop to Continue (function-scoped, auto-reverts).
-        # auto-fetch racing the deploy) can leave duplicate for-merge lines,
+        $ErrorActionPreference = 'Continue'
-        # killing the run with "Cannot fast-forward to multiple branches" even
+        git fetch origin --prune
-        # when both lines name the same commit. origin/$branch is unambiguous.
-        git fetch origin
         if ($LASTEXITCODE -ne 0) {
             throw "git fetch failed in '$root'. Check the remote, then re-run - refusing to deploy possibly-stale code."
         }
-        git merge --ff-only "origin/$branch"
+
+        # Ask the remote which branch is the default rather than assuming
+        # "main" - older repos or mirrors may differ.
+        $default = (git symbolic-ref --short refs/remotes/origin/HEAD 2>$null) -replace '^origin/', ''
+        if (-not $default) {
+            git remote set-head origin --auto | Out-Null
+            $default = (git symbolic-ref --short refs/remotes/origin/HEAD 2>$null) -replace '^origin/', ''
+        }
+        if (-not $default) { $default = 'main' }
+
+        $branch = (git rev-parse --abbrev-ref HEAD)
+        if ($branch -ne $default) {
+            # Tracked modifications only. Untracked files cannot be tangled
+            # by a branch switch, and zdeploy has always shipped them (the
+            # zip takes the working tree) - blocking on them would abort
+            # every deploy over stray local files.
+            $dirty = git status --porcelain --untracked-files=no
+            # Files the DEPLOY itself writes are excluded. zdeploy stamps the
+            # bumped build version (and appends the changelog) into the working
+            # tree after every successful run, so leaving them in scope made
+            # each deploy block the next one - the operator had to commit or
+            # stash a change they never made. The guard exists to stop
+            # unreviewed SOURCE shipping; a stamp the script just wrote is not
+            # that. It is still committed separately, one bump per PR, per the
+            # versioning rule - this only stops it being a gate.
+            $deployWritten = @('build-version.json', 'CHANGELOG.md')
+            $dirty = $dirty | Where-Object {
+                $path = ($_ -replace '^..\s+', '') -replace '^.*/', ''
+                $deployWritten -notcontains $path
+            }
+            if ($dirty) {
+                $files = ($dirty | ForEach-Object { "    $_" }) -join "`n"
+                throw "Checkout is on '$branch' with local changes:`n$files`n  Deploys ship '$default'. Commit or stash, then re-run."
+            }
+            Write-Host "  On '$branch'; deploys ship '$default' - switching." -ForegroundColor Yellow
+            git checkout $default
             if ($LASTEXITCODE -ne 0) {
-            throw "git merge --ff-only origin/$branch failed in '$root'. Resolve it (commit / stash / reconcile), then re-run - refusing to deploy possibly-stale code."
+                throw "git checkout $default failed in '$root'. Resolve it, then re-run."
+            }
+            Write-Host "  Stale branch '$branch' left in place - delete it once you've confirmed it merged." -ForegroundColor DarkGray
+        }
+
+        # Fast-forward against the remote-tracking ref, not `git pull`. The
+        # fetch at the top of this function already brought origin up to date,
+        # so pull's own fetch was redundant - and it was also the failure
+        # point: pull merges whatever FETCH_HEAD marks "for merge", and a
+        # concurrent fetch in the same repo (an editor's background auto-fetch
+        # racing the deploy) can leave duplicate for-merge lines, killing the
+        # run with "Cannot fast-forward to multiple branches" even when both
+        # lines name the same commit. origin/$default has no such ambiguity.
+        git merge --ff-only "origin/$default"
+        if ($LASTEXITCODE -ne 0) {
+            throw "git merge --ff-only origin/$default failed in '$root'. Resolve it (commit / stash / reconcile), then re-run - refusing to deploy possibly-stale code."
         }
         Write-Host "  Now at: $(git log -1 --oneline)" -ForegroundColor DarkGray
     }
@@ -604,11 +685,41 @@ if ($null -eq $global:_ZMeasuring) { $global:_ZMeasuring = $false }
 
 function Start-ZTracking {
     if ($global:_ZMeasuring) { return }
+    # Remember who is being tracked so Stop-ZTracking can log the run (ztokens).
+    try { $global:_ZTrackScript = [System.IO.Path]::GetFileNameWithoutExtension((Get-PSCallStack)[1].ScriptName) } catch { $global:_ZTrackScript = "" }
+    try {
+        $bp = (Get-PSCallStack)[1].InvocationInfo.BoundParameters
+        $global:_ZTrackProjects = (@($bp['Projects']) -join ',')
+    } catch { $global:_ZTrackProjects = "" }
     $tp = Join-Path ([System.IO.Path]::GetTempPath()) ("_ztrack_" + [System.Guid]::NewGuid().ToString("N") + ".txt")
     $global:_ZTrackPath = $tp
     try { Start-Transcript -Path $tp -NoClobber | Out-Null } catch { $global:_ZTrackPath = $null }
 }
 
+# Append this run to the ztokens data store (passive usage stats). Uses
+# $env:ZTOKENS_DATA, else the sibling ..\ztokens\data directory. Silently
+# no-ops when neither exists, so setups without ztokens are unaffected.
+function Add-ZTokensRecord {
+    param([int]$Lines, [int]$Chars, [int]$Est)
+    try {
+        $dir = $env:ZTOKENS_DATA
+        if (-not $dir) { $dir = Join-Path (Split-Path -Parent $PSScriptRoot) "ztokens\data" }
+        if (-not (Test-Path -LiteralPath $dir)) { return }
+        $model = $env:ZTOKENS_MODEL
+        if (-not $model) { $model = "est. chars/3.5" }
+        $rec = @{
+            ts       = (Get-Date).ToString("o")
+            script   = [string]$global:_ZTrackScript
+            projects = [string]$global:_ZTrackProjects
+            lines    = $Lines
+            chars    = $Chars
+            est      = $Est
+            model    = $model
+        }
+        Add-Content -LiteralPath (Join-Path $dir "tokens.jsonl") -Value (ConvertTo-Json -InputObject $rec -Compress) -Encoding UTF8
+    } catch { }
+}
+
 function Stop-ZTracking {
     if (-not $global:_ZTrackPath) { return }
     try { Stop-Transcript | Out-Null } catch {}
@@ -633,6 +744,7 @@ function Stop-ZTracking {
         $tok  = [math]::Round($cc / 3.5)
         Write-Host ""
         Write-Host ("--- {0:N0} lines / {1:N0} chars / ~{2:N0} tokens est. (Claude Code) ---" -f $lc, $cc, $tok) -ForegroundColor DarkGray
+        Add-ZTokensRecord -Lines $lc -Chars $cc -Est $tok
     } catch {}
     Remove-Item -LiteralPath $tp -Force -ErrorAction SilentlyContinue
 }

zdeploy.ps1

@@ -9,12 +9,21 @@
 #
 # Usage:
 #   zdeploy <project> [<project> ...] [-Note "message"]
-#   zdeploy all                       # every project (edge kinds first), stop at first failure
+#   zdeploy all                       # ztokens first, then every project (edge kinds next), stop at first failure
+#   zdeploy ztokens                   # refresh the live-usage stats (see below)
 #
 # Examples:
 #   zdeploy viteapp
 #   zdeploy pyapp -Note "fix billing banner"
 #   zdeploy all -Note "weekly release"
+#   zdeploy ztokens evo               # refresh token-stats.json, then ship the site with it
+#
+# "ztokens" is an OPTIONAL pseudo-project, not a zconfig entry: it runs
+# `ztokens -Publish` from a sibling ztokens checkout, if you have one, to
+# refresh a token-stats.json a site can chart. With no such checkout the step
+# prints a skip and the rest of the run is unaffected. `all` runs it first
+# automatically; called standalone, list it before a site project (as above) so
+# that project's deploy zip picks up the freshly written file.
 #
 # Flow (python/vite/nextjs): zip source -> free server disk space -> scp up ->
 # unzip into remote.path (preserving server-side .env* files and anything in
@@ -25,15 +34,12 @@
 #   python kind: app service "app", db service "db"
 #   nextjs kind: app service "web", db service "db"
 #
-# Verification (python kind when there is no scripts/build_version_tool.py, and
+# Verification (python kind, when there is no scripts/build_version_tool.py):
-# nextjs kind):
 #   Projects with a "verify" block are checked ON the server via
 #   localhost:<port><path> — the only accurate way for stacks that are not
 #   published through the edge proxy. Projects with only a "domain" fall back
 #   to a Host-header request. Projects with neither are reported as NOT
 #   verified rather than passing on the proxy's default vhost.
-#   A compose stack usually publishes to 127.0.0.1 only, so probing the public
-#   IP on the app's port can never answer — give those a "verify" block.
 #
 param(
     [Parameter(Position = 0, ValueFromRemainingArguments = $true)]
@@ -50,10 +56,10 @@ $EC2_IP     = $cfg.ec2.ip
 $PEM_KEY    = $cfg.ec2.pemKey
 $STACK_ROOT = $cfg.ec2.stackRoot
 $SSH_TARGET = Get-Ec2Target
-$RemoteHome = Get-Ec2Home
-$Ec2User    = $cfg.ec2.user
 $SSH_OPTS   = Get-Ec2SshOpts   # see ZHelpers.ps1 - these are what stop a deploy hanging
 $SCP_OPTS   = Get-Ec2ScpOpts   # same, minus -n: scp rejects it with a usage error
+$RemoteHome = Get-Ec2Home
+$Ec2User    = $cfg.ec2.user
 
 $TempRoot = $cfg.paths.temp
 if (-not (Test-Path -LiteralPath $TempRoot)) {
@@ -63,9 +69,10 @@ if (-not (Test-Path -LiteralPath $TempRoot)) {
 if ($Projects.Count -eq 0) {
     $keys = (Get-ZProjectKeys) -join ', '
     Write-Host ""
-    Write-Host "Usage: zdeploy <project> [<project> ...] | all  [-Note `"message`"]" -ForegroundColor Yellow
+    Write-Host "Usage: zdeploy <project> [<project> ...] | all | ztokens  [-Note `"message`"]" -ForegroundColor Yellow
     Write-Host "  Projects in zconfig.json: $keys" -ForegroundColor Gray
     Write-Host "  'all' deploys everything (edge kinds first) and stops at the first failure." -ForegroundColor Gray
+    Write-Host "  'ztokens' refreshes live-usage stats, if a sibling ztokens checkout exists." -ForegroundColor Gray
     Stop-ZTracking; exit 1
 }
 
@@ -73,7 +80,8 @@ if ($Projects.Count -eq 0) {
 $Projects = @($Projects | ForEach-Object { $_.TrimStart('-') })
 
 if ($Projects -contains 'all') {
-    $Projects = @(Get-ZProjectKeys)
+    # 'ztokens' first so any site project deployed below picks up fresh stats.
+    $Projects = @('ztokens') + @(Get-ZProjectKeys)
 }
 
 # Edge kinds first, however the list was produced. This is a correctness
@@ -81,7 +89,8 @@ if ($Projects -contains 'all') {
 # behind it ship, or there is a window where a new app is live behind stale
 # routing. `zdeploy evo edge` reads as "these two, edge included" and used to
 # do the risky order, because this sort only ran for 'all'.
-# Order within each group is preserved, so an intentional sequence still holds.
+# Order within each group is preserved, so an intentional sequence still holds
+# — notably `zdeploy ztokens evo`, where ztokens must still precede evo.
 $requested = @($Projects)
 $edgeKeys  = @($Projects | Where-Object { $cfg.projects.$_.kind -eq 'edge' })
 $restKeys  = @($Projects | Where-Object { $cfg.projects.$_.kind -ne 'edge' })
@@ -239,7 +248,14 @@ function Wait-VerifyApiBuild {
     param([string]$Key, $Proj, [string]$ExpectedLabel, [int]$TimeoutSec = 60)
     Write-Host "`n--- [$Key] Live build verification (expect $ExpectedLabel) ---" -ForegroundColor Cyan
     $headers = @{}
-    if ($Proj.domain) { $headers['Host'] = $Proj.domain }
+    # deploy.verifyHost overrides domain for verification only. The Host header
+    # decides which edge vhost answers, and a project's public host can be
+    # deliberately unroutable while the app is perfectly healthy - that is why
+    # the override exists. Reach for it when a domain is being retired ahead of
+    # its replacement: the old host may be returning 410 while the new one has
+    # no DNS yet, so neither answers even though the app is fine.
+    $verifyHost = if ($Proj.deploy -and $Proj.deploy.verifyHost) { $Proj.deploy.verifyHost } else { $Proj.domain }
+    if ($verifyHost) { $headers['Host'] = $verifyHost }
     $deadline = (Get-Date).AddSeconds($TimeoutSec)
     while ((Get-Date) -lt $deadline) {
         try {
@@ -526,6 +542,11 @@ function Invoke-NextDeploy {
     $prevLoc     = Get-Location
     $root        = $Proj.localRoot
     $remotePath  = $Proj.remote.path
+    # Same resolution as the python handler. Must be computed HERE: PowerShell
+    # function scope means the copy in Invoke-PythonDeploy is invisible from
+    # this one, and an unset variable interpolates to an empty string — so
+    # "cd  && docker compose down" quietly runs in the home directory.
+    $composeDir  = if ($Proj.remote.composeDir) { $Proj.remote.composeDir } else { $remotePath }
     $appSvc      = if ($Proj.remote.appService) { $Proj.remote.appService } else { "web" }
     $zipName     = Get-DeployZipName -Key $Key -Proj $Proj
     $zipLocal    = Join-Path $TempRoot $zipName
@@ -538,14 +559,39 @@ function Invoke-NextDeploy {
         Write-Host "`n=== $($Proj.label) deploy (nextjs) ===" -ForegroundColor Cyan
         Write-Host "Local zip: $zipLocal" -ForegroundColor DarkGray
 
+        # Stamp the build version from git before zipping, if the project says
+        # how (deploy.stampCmd in zconfig). The image has no .git - it is in the
+        # archive excludes - so the number has to be written on this side of the
+        # zip.
+        #
+        # Written, zipped, then reverted: zdeploy refuses a dirty tree, so a
+        # stamp that dirtied it every deploy would block the next one. The
+        # committed file stays a fallback for local dev; the number that ships
+        # is derived from the commit being deployed.
+        $stampCmd     = if ($Proj.deploy -and $Proj.deploy.stampCmd) { $Proj.deploy.stampCmd } else { $null }
+        $stampFile    = if ($Proj.deploy -and $Proj.deploy.stampFile) { $Proj.deploy.stampFile } else { "public/build-version.json" }
+        $stampedLabel = $null
+        if ($stampCmd) {
+            Write-Host "`n--- [0] Stamping build version from git ---" -ForegroundColor Cyan
+            $out = & cmd /c $stampCmd 2>&1
+            if ($LASTEXITCODE -ne 0) { throw "Build stamp failed: $out" }
+            $stampedLabel = ($out | Select-Object -Last 1).ToString().Trim()
+            Write-Host "  $stampedLabel  (derived from the commit, not a counter)" -ForegroundColor Gray
+        }
+
         $preZipBuild = Read-JsonBuildVersion -FilePath (Join-Path $root "public\build-version.json")
-        if ($preZipBuild) {
+        if ($preZipBuild -and -not $stampedLabel) {
-            Write-Host "  Pre-zip build label: $(Get-LabelFromBuildJsonObj $preZipBuild) (server-side build will bump +1)" -ForegroundColor Gray
+            Write-Host "  Pre-zip build label: $(Get-LabelFromBuildJsonObj $preZipBuild)" -ForegroundColor Gray
         }
 
         Write-Host "`n--- [1] Zipping project files ---" -ForegroundColor Cyan
         New-ProjectArchive -SourcePath $root -DestinationZip $zipLocal -TopLevelExclude (Get-ArchiveExcludes -Project $Proj)
 
+        if ($stampCmd) {
+            # Tree back to clean now the number is inside the archive.
+            git -C $root checkout -- $stampFile 2>&1 | Out-Null
+        }
+
         Invoke-Ec2PreflightCleanup -ExtraZipsToRemove @("$RemoteHome/$zipName")
 
         Write-Host "`n--- [2] Uploading zip ---" -ForegroundColor Cyan
@@ -561,16 +607,39 @@ function Invoke-NextDeploy {
         Restore-OperatorFiles -Key $Key -RemotePath $remotePath
 
         Write-Host "`n--- [4] Docker compose rebuild ---" -ForegroundColor Cyan
-        Invoke-Ec2Step "docker compose down" "cd $remotePath && sudo COMPOSE_BAKE=false docker compose down"
+        # composeDir, not remotePath: a project whose compose lives in a
-        Invoke-Ec2Step "docker compose build" "cd $remotePath && sudo COMPOSE_BAKE=false docker compose build"
+        # subdirectory (deploy/, infra/, ...) otherwise runs these against
-        Invoke-Ec2Step "docker compose up -d" "cd $remotePath && sudo COMPOSE_BAKE=false docker compose up -d"
+        # whatever docker-compose.yml happens to sit at the project root. That
+        # is usually the LOCAL DEV compose, which ships in the same archive —
+        # so the deploy recycled a dev database, found no app service to build,
+        # exited 0, and left the previous image serving. Verification passed
+        # because the untouched old container still answered. Two deploys in a
+        # row silently shipped nothing.
+        # Assert locally, not just on the server: `test -d $composeDir` with an
+        # empty value becomes bare `test -d`, which is TRUE (one non-empty
+        # argument), so the remote guard cannot catch this.
+        if ([string]::IsNullOrWhiteSpace($composeDir)) { throw "composeDir resolved empty for '$Key' - compose would run in the wrong directory." }
+        Invoke-Ec2Step "require compose file" "test -f $composeDir/docker-compose.yml || test -f $composeDir/docker-compose.yaml"
+        # BUILD BEFORE DOWN. This used to run `down` first, which took the site
+        # offline for the whole build - minutes for a Next.js app - and left it
+        # offline if the build failed. That is not hypothetical: one deploy
+        # stopped the stack, the build did not finish, and the site served 502
+        # for 19 hours with no container running at all. The
+        # old image keeps serving while the new one builds, so a failed build is
+        # now harmless and the outage is the seconds between down and up.
+        #
+        # Named service, like the python handler: a compose file that does not
+        # define it fails here instead of succeeding with nothing to do.
+        Invoke-Ec2Step "docker compose build $appSvc" "cd $composeDir && sudo COMPOSE_BAKE=false docker compose build $appSvc"
+        Invoke-Ec2Step "docker compose down" "cd $composeDir && sudo COMPOSE_BAKE=false docker compose down"
+        Invoke-Ec2Step "docker compose up -d" "cd $composeDir && sudo COMPOSE_BAKE=false docker compose up -d"
 
         if ($Proj.db -and $Proj.db.user -and $Proj.db.name) {
-            $waitDb = "cd $remotePath && for i in `$(seq 1 30); do sudo docker compose exec -T db pg_isready -U $($Proj.db.user) -d $($Proj.db.name) >/dev/null 2>&1 && break; sleep 2; done"
+            $waitDb = "cd $composeDir && for i in `$(seq 1 30); do sudo docker compose exec -T db pg_isready -U $($Proj.db.user) -d $($Proj.db.name) >/dev/null 2>&1 && break; sleep 2; done"
             Invoke-Ec2Step "wait for postgres ready" $waitDb
         }
         if ($Proj.migrations -eq "prisma") {
-            Invoke-Ec2Step "apply prisma migrations" "cd $remotePath && sudo docker compose exec -T $appSvc npx prisma migrate deploy"
+            Invoke-Ec2Step "apply prisma migrations" "cd $composeDir && sudo docker compose exec -T $appSvc npx prisma migrate deploy"
         }
         Invoke-Ec2Step "record deploy timestamp; remove remote zip" "date -u +'%Y-%m-%d %H:%M:%S UTC' | sudo tee $remotePath/.last_deploy_utc > /dev/null && rm -f $RemoteHome/$zipName"
 
@@ -604,12 +673,12 @@ function Invoke-NextDeploy {
             }
         } else {
             Write-Host "  Deploy finished - NOT verified." -ForegroundColor Yellow
-            Write-Host "  No 'domain' and no 'verify' block in zconfig.json for '$Key'," -ForegroundColor Yellow
+            Write-Host "  No 'domain' and no 'verify' block in zconfig.json for '$Key'." -ForegroundColor Yellow
-            Write-Host '  Add to the project: "verify": { "port": <hostPort>, "path": "/health" }' -ForegroundColor Gray
         }
-        if ($preZipBuild) {
+        if ($stampedLabel -or $preZipBuild) {
-            # Committed stamp, not +1 — see the note in Wait-VerifyStaticBuild.
+            # The label that actually went into the archive: the derived one
-            $expectedLabel = Get-LabelFromBuildJsonObj $preZipBuild
+            # when the project stamps, otherwise the committed stamp.
+            $expectedLabel = if ($stampedLabel) { $stampedLabel } else { Get-LabelFromBuildJsonObj $preZipBuild }
             Wait-VerifyApiBuild -Key $Key -Proj $Proj -ExpectedLabel $expectedLabel -TimeoutSec 60 | Out-Null
         } else {
             Write-Host "  (No public/build-version.json - version verification skipped. See 'Enabling deploy verification' in README.)" -ForegroundColor DarkYellow
@@ -715,9 +784,50 @@ function Invoke-DockerDeploy {
     Write-DeployLocation -Proj $Proj
 }
 
+# Pseudo-project "ztokens": not a zconfig entry, no compose stack, and entirely
+# optional. Runs `ztokens -Publish` from a sibling ztokens checkout so a site
+# that charts usage data has something current to ship. Missing checkout, or a
+# failure, warns rather than aborting the rest of the deploy list - it is a
+# nice-to-have refresh, not a deploy step.
+function Invoke-ZTokensPublish {
+    Write-Host "`n=== ztokens: refreshing live-usage stats ===" -ForegroundColor Cyan
+    $ztokensScript = Join-Path (Split-Path -Parent $PSScriptRoot) "ztokens\ztokens.ps1"
+    if (-not (Test-Path -LiteralPath $ztokensScript)) {
+        Write-Host "  ztokens.ps1 not found at $ztokensScript - skipping." -ForegroundColor Yellow
+        return
+    }
+    # Post-condition, not decoration. This step ran clean for five days while
+    # publishing nothing: ztokens.ps1 had no -Publish switch, and as a simple
+    # script PowerShell swallowed the unknown parameter into $args rather than
+    # failing. The catch below never fired because nothing threw. So the check
+    # is not "did it throw" but "did the file actually move".
+    $statsFile = Join-Path (Split-Path -Parent $PSScriptRoot) "www\public\token-stats.json"
+    $before = if (Test-Path -LiteralPath $statsFile) { (Get-Item -LiteralPath $statsFile).LastWriteTimeUtc } else { [datetime]::MinValue }
+    try {
+        & $ztokensScript -Publish
+    } catch {
+        Write-Host "  ztokens -Publish failed: $($_.Exception.Message)" -ForegroundColor Yellow
+        return
+    }
+    $after = if (Test-Path -LiteralPath $statsFile) { (Get-Item -LiteralPath $statsFile).LastWriteTimeUtc } else { [datetime]::MinValue }
+    if ($after -le $before) {
+        Write-Host "  WARNING: token-stats.json was not rewritten - the site will ship the old numbers." -ForegroundColor Yellow
+        if ($before -eq [datetime]::MinValue) {
+            Write-Host "    $statsFile does not exist." -ForegroundColor DarkGray
+        } else {
+            Write-Host ("    Still dated {0:yyyy-MM-dd HH:mm} local." -f $before.ToLocalTime()) -ForegroundColor DarkGray
+        }
+        Write-Host "    Run 'ztokens -Publish' by hand to see why." -ForegroundColor DarkGray
+    }
+}
+
 # ── Dispatch ─────────────────────────────────────────────────────────────────
 
 foreach ($key in $Projects) {
+    # 'ztokens' matches the tool it runs (ztokens.cmd / ztokens.ps1). The old
+    # singular 'ztoken' still works so existing habits and any script that
+    # already calls it keep running.
+    if ($key -in @('ztokens', 'ztoken')) { Invoke-ZTokensPublish; continue }
     $proj = Get-ZProject -Key $key
     Invoke-DeployGitPull -Proj $proj   # no-op unless deploy.gitPull is set
     switch ([string]$proj.kind) {